Oct 30 20:35:55.985616 ip-10-0-131-125 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Oct 30 20:35:55.985627 ip-10-0-131-125 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Oct 30 20:35:55.985636 ip-10-0-131-125 systemd[1]: kubelet.service: Failed with result 'resources'.
Oct 30 20:35:55.985939 ip-10-0-131-125 systemd[1]: Failed to start Kubernetes Kubelet.
Oct 30 20:36:07.347198 ip-10-0-131-125 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Oct 30 20:36:07.347214 ip-10-0-131-125 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot d8c270a2868549aea7de36fcea22edd0 --
Oct 30 20:38:11.712600 ip-10-0-131-125 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 20:38:12.089725 ip-10-0-131-125 kubenswrapper[2583]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 20:38:12.089725 ip-10-0-131-125 kubenswrapper[2583]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 20:38:12.089725 ip-10-0-131-125 kubenswrapper[2583]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 20:38:12.089725 ip-10-0-131-125 kubenswrapper[2583]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 20:38:12.096178 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.096090    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 20:38:12.096178 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.096178    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 20:38:12.096178 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.096183    2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 20:38:12.096280 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.096186    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 20:38:12.096280 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.096190    2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 20:38:12.096280 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.096193    2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 20:38:12.096280 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.096195    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 20:38:12.096280 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.096198    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 20:38:12.096280 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.096201    2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 20:38:12.096280 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.096203    2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 20:38:12.096280 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.096206    2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 20:38:12.096280 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.096209    2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 20:38:12.096280 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.096212    2583 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 20:38:12.096280 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.096216    2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 20:38:12.096280 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.096220    2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 20:38:12.096280 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.096223    2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 20:38:12.096280 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.096226    2583 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 20:38:12.096280 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.096229    2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 20:38:12.096280 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.096232    2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 20:38:12.096280 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.096235    2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 20:38:12.096280 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.096237    2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 20:38:12.096280 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.096240    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 20:38:12.096280 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.096243    2583 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 20:38:12.096763 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.096246    2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 20:38:12.096763 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.096249    2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 20:38:12.096763 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.096251    2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 20:38:12.096763 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.096254    2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 20:38:12.096763 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.096260    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 20:38:12.096763 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.096263    2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 20:38:12.096763 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.096266    2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 20:38:12.096763 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.096268    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 20:38:12.096763 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.096272    2583 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 20:38:12.096763 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.096274    2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 20:38:12.096763 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.096277    2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 20:38:12.096763 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.096280    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 20:38:12.096763 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.096283    2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 20:38:12.096763 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.096287    2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 20:38:12.096763 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.096289    2583 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 20:38:12.096763 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.096292    2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 20:38:12.096763 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.096295    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 20:38:12.096763 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.096297    2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 20:38:12.096763 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.096300    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 20:38:12.097196 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.096303    2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 20:38:12.097196 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.096305    2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 20:38:12.097196 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.096308    2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 20:38:12.097196 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.096312    2583 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 20:38:12.097196 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.096314    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 20:38:12.097196 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.096317    2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 20:38:12.097196 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.096319    2583 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 20:38:12.097196 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.096333    2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 20:38:12.097196 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.096336    2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 20:38:12.097196 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.096339    2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 20:38:12.097196 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.096342    2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 20:38:12.097196 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.096345    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 20:38:12.097196 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.096347    2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 20:38:12.097196 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.096350    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 20:38:12.097196 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.096353    2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 20:38:12.097196 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.096356    2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 20:38:12.097196 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.096358    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 20:38:12.097196 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.096361    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 20:38:12.097196 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.096364    2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 20:38:12.097652 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.096368    2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 20:38:12.097652 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.096370    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 20:38:12.097652 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.096373    2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 20:38:12.097652 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.096376    2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 20:38:12.097652 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.096378    2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 20:38:12.097652 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.096381    2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 20:38:12.097652 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.096384    2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 20:38:12.097652 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.096386    2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 20:38:12.097652 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.096389    2583 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 20:38:12.097652 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.096391    2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 20:38:12.097652 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.096394    2583 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 20:38:12.097652 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.096396    2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 20:38:12.097652 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.096399    2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 20:38:12.097652 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.096401    2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 20:38:12.097652 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.096404    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 20:38:12.097652 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.096406    2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 20:38:12.097652 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.096409    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 20:38:12.097652 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.096412    2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 20:38:12.097652 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.096414    2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 20:38:12.097652 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.096418    2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 20:38:12.098130 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.096421    2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 20:38:12.098130 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.096423    2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 20:38:12.098130 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.096426    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 20:38:12.098130 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.096429    2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 20:38:12.098130 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.096431    2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 20:38:12.098130 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.096434    2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 20:38:12.098130 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.096437    2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 20:38:12.098130 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.096439    2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 20:38:12.098130 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.096442    2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 20:38:12.098130 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.096444    2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 20:38:12.098130 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.096447    2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 20:38:12.098130 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.096449    2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 20:38:12.098130 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.096451    2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 20:38:12.098130 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.096454    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 20:38:12.098130 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.096457    2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 20:38:12.098130 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.096459    2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 20:38:12.098130 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.096462    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 20:38:12.098130 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.096464    2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 20:38:12.098130 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.096468    2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 20:38:12.098130 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.096472    2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 20:38:12.098698 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.096475    2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 20:38:12.098698 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.096478    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 20:38:12.098698 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.096482    2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 20:38:12.098698 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.096484    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 20:38:12.098698 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.096487    2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 20:38:12.098698 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.098166    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 20:38:12.098698 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.098174    2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 20:38:12.098698 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.098178    2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 20:38:12.098698 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.098181    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 20:38:12.098698 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.098184    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 20:38:12.098698 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.098193    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 20:38:12.098698 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.098196    2583 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 20:38:12.098698 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.098198    2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 20:38:12.098698 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.098201    2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 20:38:12.098698 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.098205    2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 20:38:12.098698 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.098209    2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 20:38:12.098698 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.098212    2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 20:38:12.098698 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.098214    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 20:38:12.098698 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.098217    2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 20:38:12.099166 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.098220    2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 20:38:12.099166 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.098222    2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 20:38:12.099166 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.098225    2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 20:38:12.099166 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.098227    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 20:38:12.099166 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.098230    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 20:38:12.099166 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.098233    2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 20:38:12.099166 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.098235    2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 20:38:12.099166 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.098238    2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 20:38:12.099166 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.098241    2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 20:38:12.099166 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.098243    2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 20:38:12.099166 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.098245    2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 20:38:12.099166 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.098248    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 20:38:12.099166 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.098251    2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 20:38:12.099166 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.098253    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 20:38:12.099166 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.098256    2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 20:38:12.099166 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.098259    2583 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 20:38:12.099166 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.098262    2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 20:38:12.099166 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.098265    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 20:38:12.099166 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.098268    2583 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 20:38:12.099166 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.098270    2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 20:38:12.099656 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.098273    2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 20:38:12.099656 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.098275    2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 20:38:12.099656 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.098278    2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 20:38:12.099656 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.098280    2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 20:38:12.099656 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.098284    2583 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 20:38:12.099656 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.098287    2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 20:38:12.099656 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.098289    2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 20:38:12.099656 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.098292    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 20:38:12.099656 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.098295    2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 20:38:12.099656 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.098297    2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 20:38:12.099656 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.098299    2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 20:38:12.099656 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.098302    2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 20:38:12.099656 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.098305    2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 20:38:12.099656 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.098307    2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 20:38:12.099656 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.098310    2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 20:38:12.099656 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.098312    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 20:38:12.099656 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.098315    2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 20:38:12.099656 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.098319    2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 20:38:12.099656 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.098334    2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 20:38:12.100137 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.098337    2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 20:38:12.100137 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.098340    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 20:38:12.100137 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.098343    2583 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 20:38:12.100137 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.098346    2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 20:38:12.100137 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.098349    2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 20:38:12.100137 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.098351    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 20:38:12.100137 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.098356    2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 20:38:12.100137 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.098359    2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 20:38:12.100137 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.098362    2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 20:38:12.100137 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.098364    2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 20:38:12.100137 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.098367    2583 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 20:38:12.100137 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.098370    2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 20:38:12.100137 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.098372    2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 20:38:12.100137 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.098375    2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 20:38:12.100137 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.098378    2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 20:38:12.100137 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.098381    2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 20:38:12.100137 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.098383    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 20:38:12.100137 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.098386    2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 20:38:12.100137 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.098389    2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 20:38:12.100137 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.098391    2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 20:38:12.100626 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.098394    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 20:38:12.100626 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.098396    2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 20:38:12.100626 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.098399    2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 20:38:12.100626 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.098402    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 20:38:12.100626 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.098404    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 20:38:12.100626 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.098407    2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 20:38:12.100626 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.098409    2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 20:38:12.100626 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.098412    2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 20:38:12.100626 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.098414    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 20:38:12.100626 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.098417    2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 20:38:12.100626 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.098419    2583 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 20:38:12.100626 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.098422    2583 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 20:38:12.100626 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.098424    2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 20:38:12.100626 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.098427    2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 20:38:12.100626 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.098430    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 20:38:12.100626 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.098432    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 20:38:12.100626 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.098435    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 20:38:12.100626 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.098438    2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 20:38:12.100626 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.098441    2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 20:38:12.100626 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.098444    2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 20:38:12.101110 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.098446    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 20:38:12.101110 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.098449    2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 20:38:12.101110 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.098452    2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 20:38:12.101110 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.098454    2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 20:38:12.101110 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.098457    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 20:38:12.101110 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.098460    2583 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 20:38:12.101110 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.098463    2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 20:38:12.101110 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.098465    2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 20:38:12.101110 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.098468    2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 20:38:12.101110 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.098470    2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 20:38:12.101110 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.098473    2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 20:38:12.101110 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.098477    2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 20:38:12.101110 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.098480    2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 20:38:12.101110 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.099576    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 20:38:12.101110 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.099583    2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 20:38:12.101110 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.099586    2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 20:38:12.101110 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.099589    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 20:38:12.101110 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.099592    2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 20:38:12.101110 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.099594    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 20:38:12.101565 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.099597    2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 20:38:12.101565 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.099600    2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 20:38:12.101565 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.099602    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 20:38:12.101565 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.099605    2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 20:38:12.101565 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.099608    2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 20:38:12.101565 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.099610    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 20:38:12.101565 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.099613    2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 20:38:12.101565 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.099616    2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 20:38:12.101565 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.099619    2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 20:38:12.101565 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.099622    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 20:38:12.101565 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.099626    2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 20:38:12.101565 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.099628    2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 20:38:12.101565 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.099631    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 20:38:12.101565 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.099634    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 20:38:12.101565 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.099637    2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 20:38:12.101565 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.099639    2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 20:38:12.101565 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.099642    2583 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 20:38:12.101565 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.099644    2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 20:38:12.101565 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.099647    2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 20:38:12.101565 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.099649    2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 20:38:12.102041 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.099652    2583 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 20:38:12.102041 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.099655    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 20:38:12.102041 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.099657    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 20:38:12.102041 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.099660    2583 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 20:38:12.102041 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.099663    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 20:38:12.102041 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.099666    2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 20:38:12.102041 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.099669    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 20:38:12.102041 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.099672    2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 20:38:12.102041 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.099674    2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 20:38:12.102041 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.099677    2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 20:38:12.102041 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.099680    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 20:38:12.102041 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.099682    2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 20:38:12.102041 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.099685    2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 20:38:12.102041 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.099687    2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 20:38:12.102041 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.099690    2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 20:38:12.102041 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.099693    2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 20:38:12.102041 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.099695    2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 20:38:12.102041 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.099698    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 20:38:12.102041 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.099700    2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 20:38:12.102041 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.099703    2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 20:38:12.102562 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.099705    2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 20:38:12.102562 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.099708    2583 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 20:38:12.102562 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.099711    2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 20:38:12.102562 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.099713    2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 20:38:12.102562 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.099716    2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 20:38:12.102562 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.099719    2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 20:38:12.102562 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.099721    2583 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 20:38:12.102562 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.099726    2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 20:38:12.102562 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.099729    2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 20:38:12.102562 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.099732    2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 20:38:12.102562 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.099734    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 20:38:12.102562 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.099737    2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 20:38:12.102562 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.099740    2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 20:38:12.102562 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.099742    2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 20:38:12.102562 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.099745    2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 20:38:12.102562 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.099747    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 20:38:12.102562 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.099750    2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 20:38:12.102562 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.099753    2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 20:38:12.102562 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.099755    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 20:38:12.103024 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.099757    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 20:38:12.103024 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.099760    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 20:38:12.103024 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.099762    2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 20:38:12.103024 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.099765    2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 20:38:12.103024 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.099767    2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 20:38:12.103024 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.099770    2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 20:38:12.103024 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.099772    2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 20:38:12.103024 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.099775    2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 20:38:12.103024 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.099781    2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 20:38:12.103024 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.099785    2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 20:38:12.103024 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.099789    2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 20:38:12.103024 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.099792    2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 20:38:12.103024 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.099795    2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 20:38:12.103024 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.099797    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 20:38:12.103024 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.099800    2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 20:38:12.103024 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.099803    2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 20:38:12.103024 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.099805    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 20:38:12.103024 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.099808    2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 20:38:12.103024 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.099811    2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 20:38:12.103477 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.099813    2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 20:38:12.103477 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.099816    2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 20:38:12.103477 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.099819    2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 20:38:12.103477 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.099822    2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 20:38:12.103477 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.099824    2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 20:38:12.103477 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.099827    2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 20:38:12.103477 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.099829    2583 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 20:38:12.103477 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.099832    2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 20:38:12.103477 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.099835    2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 20:38:12.103477 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.099837    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 20:38:12.103477 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.099840    2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 20:38:12.103477 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.099842    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 20:38:12.103477 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.099845    2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 20:38:12.103477 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.099848    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 20:38:12.103477 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.099850    2583 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 20:38:12.103477 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.099853    2583 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 20:38:12.103477 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.099856    2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 20:38:12.103477 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.099858    2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 20:38:12.103477 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.099862    2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 20:38:12.103477 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.099864    2583 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 20:38:12.103968 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.099867    2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 20:38:12.103968 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.099871    2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 20:38:12.103968 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.101145    2583 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 20:38:12.103968 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.101155    2583 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 20:38:12.103968 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.101161    2583 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 20:38:12.103968 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.101166    2583 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 20:38:12.103968 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.101172    2583 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 20:38:12.103968 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.101175    2583 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 20:38:12.103968 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.101180    2583 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 20:38:12.103968 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.101184    2583 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 20:38:12.103968 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.101188    2583 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 20:38:12.103968 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.101191    2583 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 20:38:12.103968 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.101195    2583 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 20:38:12.103968 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.101199    2583 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 20:38:12.103968 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.101203    2583 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 20:38:12.103968 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.101206    2583 flags.go:64] FLAG: --cgroup-root=""
Oct 30 20:38:12.103968 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.101209    2583 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 20:38:12.103968 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.101212    2583 flags.go:64] FLAG: --client-ca-file=""
Oct 30 20:38:12.103968 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.101215    2583 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 20:38:12.103968 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.101218    2583 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 20:38:12.103968 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.101223    2583 flags.go:64] FLAG: --cluster-domain=""
Oct 30 20:38:12.103968 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.101226    2583 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 20:38:12.103968 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.101229    2583 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 20:38:12.103968 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.101232    2583 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 20:38:12.104524 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.101236    2583 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 20:38:12.104524 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.101241    2583 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 20:38:12.104524 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.101244    2583 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 20:38:12.104524 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.101247    2583 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 20:38:12.104524 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.101250    2583 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 20:38:12.104524 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.101253    2583 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 20:38:12.104524 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.101257    2583 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 20:38:12.104524 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.101260    2583 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 20:38:12.104524 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.101263    2583 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 20:38:12.104524 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.101267    2583 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 20:38:12.104524 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.101271    2583 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 20:38:12.104524 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.101275    2583 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 20:38:12.104524 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.101278    2583 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 20:38:12.104524 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.101281    2583 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 20:38:12.104524 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.101284    2583 flags.go:64] FLAG: --enable-server="true"
Oct 30 20:38:12.104524 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.101287    2583 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 20:38:12.104524 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.101291    2583 flags.go:64] FLAG: --event-burst="100"
Oct 30 20:38:12.104524 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.101295    2583 flags.go:64] FLAG: --event-qps="50"
Oct 30 20:38:12.104524 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.101298    2583 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 20:38:12.104524 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.101301    2583 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 20:38:12.104524 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.101304    2583 flags.go:64] FLAG: --eviction-hard=""
Oct 30 20:38:12.104524 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.101309    2583 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 20:38:12.104524 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.101312    2583 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 20:38:12.104524 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.101315    2583 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 20:38:12.104524 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.101318    2583 flags.go:64] FLAG: --eviction-soft=""
Oct 30 20:38:12.105131 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.101332    2583 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 20:38:12.105131 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.101336    2583 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 20:38:12.105131 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.101339    2583 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 20:38:12.105131 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.101342    2583 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 20:38:12.105131 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.101345    2583 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 20:38:12.105131 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.101348    2583 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 20:38:12.105131 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.101351    2583 flags.go:64] FLAG: --feature-gates=""
Oct 30 20:38:12.105131 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.101355    2583 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 20:38:12.105131 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.101358    2583 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 20:38:12.105131 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.101362    2583 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 20:38:12.105131 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.101366    2583 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 20:38:12.105131 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.101369    2583 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 20:38:12.105131 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.101372    2583 flags.go:64] FLAG: --help="false"
Oct 30 20:38:12.105131 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.101375    2583 flags.go:64] FLAG: --hostname-override="ip-10-0-131-125.us-east-2.compute.internal"
Oct 30 20:38:12.105131 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.101379    2583 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 20:38:12.105131 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.101382    2583 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 20:38:12.105131 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.101385    2583 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 20:38:12.105131 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.101388    2583 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 20:38:12.105131 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.101392    2583 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 20:38:12.105131 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.101395    2583 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 20:38:12.105131 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.101398    2583 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 20:38:12.105131 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.101401    2583 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 20:38:12.105131 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.101403    2583 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 20:38:12.105725 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.101409    2583 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 20:38:12.105725 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.101412    2583 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 20:38:12.105725 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.101415    2583 flags.go:64] FLAG: --kube-reserved=""
Oct 30 20:38:12.105725 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.101418    2583 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 20:38:12.105725 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.101421    2583 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 20:38:12.105725 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.101425    2583 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 20:38:12.105725 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.101427    2583 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 20:38:12.105725 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.101430    2583 flags.go:64] FLAG: --lock-file=""
Oct 30 20:38:12.105725 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.101433    2583 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 20:38:12.105725 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.101436    2583 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 20:38:12.105725 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.101439    2583 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 20:38:12.105725 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.101445    2583 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 20:38:12.105725 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.101448    2583 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 20:38:12.105725 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.101451    2583 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 20:38:12.105725 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.101454    2583 flags.go:64] FLAG: --logging-format="text"
Oct 30 20:38:12.105725 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.101457    2583 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 20:38:12.105725 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.101461    2583 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 20:38:12.105725 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.101464    2583 flags.go:64] FLAG: --manifest-url=""
Oct 30 20:38:12.105725 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.101466    2583 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 20:38:12.105725 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.101471    2583 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 20:38:12.105725 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.101474    2583 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 20:38:12.105725 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.101478    2583 flags.go:64] FLAG: --max-pods="110"
Oct 30 20:38:12.105725 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.101481    2583 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 20:38:12.105725 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.101484    2583 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 20:38:12.105725 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.101487    2583 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 20:38:12.106357 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.101490    2583 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 20:38:12.106357 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.101493    2583 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 20:38:12.106357 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.101496    2583 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 20:38:12.106357 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.101499    2583 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 20:38:12.106357 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.101506    2583 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 20:38:12.106357 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.101510    2583 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 20:38:12.106357 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.101513    2583 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 20:38:12.106357 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.101517    2583 flags.go:64] FLAG: --pod-cidr=""
Oct 30 20:38:12.106357 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.101520    2583 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 20:38:12.106357 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.101523    2583 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 20:38:12.106357 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.101527    2583 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 20:38:12.106357 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.101530    2583 flags.go:64] FLAG: --port="10250"
Oct 30 20:38:12.106357 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.101533    2583 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 20:38:12.106357 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.101536    2583 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-0775184a58f4035a6"
Oct 30 20:38:12.106357 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.101539    2583 flags.go:64] FLAG: --qos-reserved=""
Oct 30 20:38:12.106357 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.101542    2583 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 20:38:12.106357 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.101545    2583 flags.go:64] FLAG: --register-node="true"
Oct 30 20:38:12.106357 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.101548    2583 flags.go:64] FLAG: --register-with-taints=""
Oct 30 20:38:12.106357 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.101552    2583 flags.go:64] FLAG: --registry-burst="10"
Oct 30 20:38:12.106357 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.101555    2583 flags.go:64] FLAG: --registry-qps="5"
Oct 30 20:38:12.106357 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.101558    2583 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 20:38:12.106357 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.101561    2583 flags.go:64] FLAG: --reserved-memory=""
Oct 30 20:38:12.106357 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.101565    2583 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 20:38:12.106357 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.101568    2583 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 20:38:12.106357 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.101572    2583 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 20:38:12.106357 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.101575    2583 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 20:38:12.106982 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.101578    2583 flags.go:64] FLAG: --runonce="false"
Oct 30 20:38:12.106982 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.101582    2583 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 20:38:12.106982 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.101585    2583 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 20:38:12.106982 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.101588    2583 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 20:38:12.106982 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.101591    2583 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 20:38:12.106982 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.101594    2583 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 20:38:12.106982 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.101597    2583 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 20:38:12.106982 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.101600    2583 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 20:38:12.106982 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.101603    2583 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 20:38:12.106982 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.101606    2583 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 20:38:12.106982 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.101609    2583 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 20:38:12.106982 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.101612    2583 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 20:38:12.106982 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.101615    2583 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 20:38:12.106982 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.101619    2583 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 20:38:12.106982 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.101622    2583 flags.go:64] FLAG: --system-cgroups=""
Oct 30 20:38:12.106982 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.101624    2583 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 20:38:12.106982 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.101630    2583 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 20:38:12.106982 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.101633    2583 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 20:38:12.106982 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.101636    2583 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 20:38:12.106982 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.101641    2583 flags.go:64] FLAG: --tls-min-version=""
Oct 30 20:38:12.106982 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.101644    2583 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 20:38:12.106982 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.101646    2583 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 20:38:12.106982 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.101650    2583 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 20:38:12.106982 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.101653    2583 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 20:38:12.106982 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.101655    2583 flags.go:64] FLAG: --v="2"
Oct 30 20:38:12.107626 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.101661    2583 flags.go:64] FLAG: --version="false"
Oct 30 20:38:12.107626 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.101666    2583 flags.go:64] FLAG: --vmodule=""
Oct 30 20:38:12.107626 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.101670    2583 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 20:38:12.107626 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.101673    2583 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 20:38:12.107626 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.101818    2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 20:38:12.107626 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.101822    2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 20:38:12.107626 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.101825    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 20:38:12.107626 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.101828    2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 20:38:12.107626 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.101831    2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 20:38:12.107626 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.101834    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 20:38:12.107626 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.101837    2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 20:38:12.107626 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.101839    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 20:38:12.107626 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.101842    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 20:38:12.107626 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.101845    2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 20:38:12.107626 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.101847    2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 20:38:12.107626 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.101850    2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 20:38:12.107626 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.101852    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 20:38:12.107626 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.101855    2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 20:38:12.107626 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.101858    2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 20:38:12.107626 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.101860    2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 20:38:12.108096 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.101863    2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 20:38:12.108096 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.101866    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 20:38:12.108096 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.101868    2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 20:38:12.108096 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.101871    2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 20:38:12.108096 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.101873    2583 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 20:38:12.108096 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.101876    2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 20:38:12.108096 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.101878    2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 20:38:12.108096 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.101881    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 20:38:12.108096 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.101884    2583 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 20:38:12.108096 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.101887    2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 20:38:12.108096 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.101890    2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 20:38:12.108096 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.101892    2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 20:38:12.108096 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.101896    2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 20:38:12.108096 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.101899    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 20:38:12.108096 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.101902    2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 20:38:12.108096 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.101904    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 20:38:12.108096 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.101907    2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 20:38:12.108096 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.101910    2583 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 20:38:12.108096 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.101912    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 20:38:12.108096 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.101915    2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 20:38:12.108629 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.101918    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 20:38:12.108629 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.101921    2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 20:38:12.108629 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.101924    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 20:38:12.108629 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.101926    2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 20:38:12.108629 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.101929    2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 20:38:12.108629 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.101931    2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 20:38:12.108629 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.101934    2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 20:38:12.108629 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.101937    2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 20:38:12.108629 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.101939    2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 20:38:12.108629 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.101942    2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 20:38:12.108629 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.101944    2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 20:38:12.108629 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.101947    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 20:38:12.108629 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.101949    2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 20:38:12.108629 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.101952    2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 20:38:12.108629 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.101954    2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 20:38:12.108629 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.101957    2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 20:38:12.108629 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.101959    2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 20:38:12.108629 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.101962    2583 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 20:38:12.108629 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.101964    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 20:38:12.108629 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.101967    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 20:38:12.109096 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.101969    2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 20:38:12.109096 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.101972    2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 20:38:12.109096 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.101975    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 20:38:12.109096 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.101977    2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 20:38:12.109096 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.101981    2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 20:38:12.109096 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.101983    2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 20:38:12.109096 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.101986    2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 20:38:12.109096 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.101988    2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 20:38:12.109096 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.101991    2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 20:38:12.109096 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.101994    2583 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 20:38:12.109096 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.101996    2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 20:38:12.109096 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.101999    2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 20:38:12.109096 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.102003    2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 20:38:12.109096 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.102007    2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 20:38:12.109096 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.102009    2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 20:38:12.109096 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.102013    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 20:38:12.109096 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.102015    2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 20:38:12.109096 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.102018    2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 20:38:12.109096 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.102020    2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 20:38:12.109562 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.102023    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 20:38:12.109562 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.102025    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 20:38:12.109562 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.102028    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 20:38:12.109562 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.102030    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 20:38:12.109562 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.102033    2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 20:38:12.109562 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.102035    2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 20:38:12.109562 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.102038    2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 20:38:12.109562 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.102041    2583 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 20:38:12.109562 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.102043    2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 20:38:12.109562 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.102046    2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 20:38:12.109562 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.102049    2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 20:38:12.109562 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.102052    2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 20:38:12.109562 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.102056    2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 20:38:12.109562 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.102059    2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 20:38:12.109562 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.102062    2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 20:38:12.109562 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.102064    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 20:38:12.109562 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.102067    2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 20:38:12.109562 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.102070    2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 20:38:12.109562 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.102073    2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 20:38:12.110012 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.102076    2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 20:38:12.110012 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.102078    2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 20:38:12.110012 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.102081    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 20:38:12.110012 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.102084    2583 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 20:38:12.110012 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.102086    2583 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 20:38:12.110012 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.102089    2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 20:38:12.110012 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.102091    2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 20:38:12.110012 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.102094    2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 20:38:12.110012 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.102097    2583 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 20:38:12.110012 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.102099    2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 20:38:12.110012 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.102102    2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 20:38:12.110012 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.102105    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 20:38:12.110012 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.102131    2583 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 20:38:12.110012 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.102174    2583 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 20:38:12.110012 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.102179    2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 20:38:12.110012 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.102182    2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 20:38:12.110402 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.102187    2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 20:38:12.110402 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.102190    2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 20:38:12.110402 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.102193    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 20:38:12.110402 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.102196    2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 20:38:12.110402 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.102199    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 20:38:12.110402 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.102202    2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 20:38:12.110402 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.102204    2583 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 20:38:12.110402 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.102207    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 20:38:12.110402 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.102210    2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 20:38:12.110402 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.102212    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 20:38:12.110402 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.102215    2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 20:38:12.110402 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.102218    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 20:38:12.110402 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.102220    2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 20:38:12.110402 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.102223    2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 20:38:12.110402 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.102225    2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 20:38:12.110402 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.102229    2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 20:38:12.110402 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.102232    2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 20:38:12.110402 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.102234    2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 20:38:12.110402 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.102237    2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 20:38:12.110402 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.102239    2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 20:38:12.110877 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.102242    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 20:38:12.110877 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.102245    2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 20:38:12.110877 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.102247    2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 20:38:12.110877 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.102250    2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 20:38:12.110877 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.102253    2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 20:38:12.110877 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.102255    2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 20:38:12.110877 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.102258    2583 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 20:38:12.110877 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.102261    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 20:38:12.110877 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.102264    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 20:38:12.110877 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.102266    2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 20:38:12.110877 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.102269    2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 20:38:12.110877 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.102272    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 20:38:12.110877 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.102275    2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 20:38:12.110877 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.102278    2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 20:38:12.110877 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.102281    2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 20:38:12.110877 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.102284    2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 20:38:12.110877 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.102286    2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 20:38:12.110877 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.102289    2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 20:38:12.110877 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.102291    2583 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 20:38:12.110877 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.102294    2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 20:38:12.111372 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.102297    2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 20:38:12.111372 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.102301    2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 20:38:12.111372 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.102304    2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 20:38:12.111372 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.102307    2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 20:38:12.111372 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.102311    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 20:38:12.111372 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.102313    2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 20:38:12.111372 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.102316    2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 20:38:12.111372 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.102331    2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 20:38:12.111372 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.102334    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 20:38:12.111372 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.102337    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 20:38:12.111372 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.102340    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 20:38:12.111372 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.102343    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 20:38:12.111372 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.102345    2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 20:38:12.111372 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.102348    2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 20:38:12.111372 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.102350    2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 20:38:12.111372 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.102353    2583 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 20:38:12.111372 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.102355    2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 20:38:12.111372 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.102358    2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 20:38:12.111372 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.102361    2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 20:38:12.111828 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.102365    2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 20:38:12.111828 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.102369    2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 20:38:12.111828 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.102373    2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 20:38:12.111828 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.102376    2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 20:38:12.111828 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.102379    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 20:38:12.111828 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.102382    2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 20:38:12.111828 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.102385    2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 20:38:12.111828 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.102388    2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 20:38:12.111828 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.102390    2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 20:38:12.111828 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.102393    2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 20:38:12.111828 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.102396    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 20:38:12.111828 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.102398    2583 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 20:38:12.111828 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.102401    2583 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 20:38:12.111828 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.102403    2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 20:38:12.111828 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.102406    2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 20:38:12.111828 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.102408    2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 20:38:12.111828 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.102411    2583 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 20:38:12.111828 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.102414    2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 20:38:12.111828 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.102416    2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 20:38:12.112261 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.102419    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 20:38:12.112261 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.102423    2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 20:38:12.112261 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.102426    2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 20:38:12.112261 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.102428    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 20:38:12.112261 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.102431    2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 20:38:12.112261 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.102434    2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 20:38:12.112261 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.102436    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 20:38:12.112261 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.102439    2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 20:38:12.112261 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.102441    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 20:38:12.112261 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.102444    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 20:38:12.112261 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.102446    2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 20:38:12.112261 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.102449    2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 20:38:12.112261 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.102452    2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 20:38:12.112261 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.102454    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 20:38:12.112261 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.102457    2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 20:38:12.112261 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.102460    2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 20:38:12.112261 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.102463    2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 20:38:12.112261 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.102466    2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 20:38:12.112261 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.102469    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 20:38:12.112261 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.102472    2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 20:38:12.112756 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.102474    2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 20:38:12.112756 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.102477    2583 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 20:38:12.112756 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.102479    2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 20:38:12.112756 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.102482    2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 20:38:12.112756 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.102485    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 20:38:12.112756 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.109528    2583 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 20:38:12.112756 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.109550    2583 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 20:38:12.112756 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.109566    2583 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 20:38:12.112756 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.109571    2583 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 20:38:12.112756 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.109636    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 20:38:12.112756 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.109640    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 20:38:12.112756 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.109644    2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 20:38:12.112756 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.109646    2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 20:38:12.112756 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.109649    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 20:38:12.112756 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.109652    2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 20:38:12.112756 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.109655    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 20:38:12.112756 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.109657    2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 20:38:12.112756 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.109660    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 20:38:12.112756 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.109663    2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 20:38:12.113198 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.109665    2583 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 20:38:12.113198 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.109668    2583 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 20:38:12.113198 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.109670    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 20:38:12.113198 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.109673    2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 20:38:12.113198 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.109676    2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 20:38:12.113198 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.109679    2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 20:38:12.113198 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.109681    2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 20:38:12.113198 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.109684    2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 20:38:12.113198 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.109687    2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 20:38:12.113198 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.109690    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 20:38:12.113198 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.109692    2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 20:38:12.113198 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.109695    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 20:38:12.113198 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.109697    2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 20:38:12.113198 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.109700    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 20:38:12.113198 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.109703    2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 20:38:12.113198 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.109705    2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 20:38:12.113198 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.109708    2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 20:38:12.113198 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.109710    2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 20:38:12.113198 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.109713    2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 20:38:12.113198 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.109722    2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 20:38:12.113681 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.109725    2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 20:38:12.113681 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.109728    2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 20:38:12.113681 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.109730    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 20:38:12.113681 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.109733    2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 20:38:12.113681 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.109736    2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 20:38:12.113681 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.109738    2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 20:38:12.113681 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.109741    2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 20:38:12.113681 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.109743    2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 20:38:12.113681 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.109746    2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 20:38:12.113681 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.109748    2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 20:38:12.113681 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.109751    2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 20:38:12.113681 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.109753    2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 20:38:12.113681 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.109756    2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 20:38:12.113681 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.109759    2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 20:38:12.113681 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.109761    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 20:38:12.113681 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.109764    2583 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 20:38:12.113681 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.109766    2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 20:38:12.113681 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.109769    2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 20:38:12.113681 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.109772    2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 20:38:12.113681 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.109774    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 20:38:12.114176 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.109777    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 20:38:12.114176 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.109780    2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 20:38:12.114176 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.109784    2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 20:38:12.114176 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.109788    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 20:38:12.114176 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.109791    2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 20:38:12.114176 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.109794    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 20:38:12.114176 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.109797    2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 20:38:12.114176 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.109800    2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 20:38:12.114176 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.109803    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 20:38:12.114176 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.109806    2583 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 20:38:12.114176 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.109808    2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 20:38:12.114176 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.109812    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 20:38:12.114176 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.109820    2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 20:38:12.114176 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.109823    2583 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 20:38:12.114176 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.109826    2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 20:38:12.114176 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.109829    2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 20:38:12.114176 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.109834    2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 20:38:12.114176 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.109836    2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 20:38:12.114176 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.109839    2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 20:38:12.114685 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.109842    2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 20:38:12.114685 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.109844    2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 20:38:12.114685 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.109847    2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 20:38:12.114685 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.109850    2583 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 20:38:12.114685 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.109852    2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 20:38:12.114685 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.109855    2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 20:38:12.114685 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.109857    2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 20:38:12.114685 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.109860    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 20:38:12.114685 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.109862    2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 20:38:12.114685 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.109865    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 20:38:12.114685 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.109867    2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 20:38:12.114685 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.109870    2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 20:38:12.114685 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.109872    2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 20:38:12.114685 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.109875    2583 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 20:38:12.114685 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.109878    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 20:38:12.114685 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.109880    2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 20:38:12.114685 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.109882    2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 20:38:12.114685 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.109885    2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 20:38:12.114685 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.109887    2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 20:38:12.114685 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.109890    2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 20:38:12.115173 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.109892    2583 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 20:38:12.115173 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.109895    2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 20:38:12.115173 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.109897    2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 20:38:12.115173 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.109900    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 20:38:12.115173 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.109903    2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 20:38:12.115173 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.109906    2583 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 20:38:12.115173 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.109914    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 20:38:12.115173 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.109917    2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 20:38:12.115173 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.109920    2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 20:38:12.115173 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.109922    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 20:38:12.115173 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.109925    2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 20:38:12.115173 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.109927    2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 20:38:12.115173 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.109930    2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 20:38:12.115173 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.109933    2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 20:38:12.115173 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.109935    2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 20:38:12.115173 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.109937    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 20:38:12.115173 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.109940    2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 20:38:12.115608 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.109968    2583 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 20:38:12.115608 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110165    2583 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 20:38:12.115608 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110169    2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 20:38:12.115608 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110173    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 20:38:12.115608 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110176    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 20:38:12.115608 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110178    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 20:38:12.115608 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110181    2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 20:38:12.115608 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110184    2583 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 20:38:12.115608 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110186    2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 20:38:12.115608 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110189    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 20:38:12.115608 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110192    2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 20:38:12.115608 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110194    2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 20:38:12.115608 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110197    2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 20:38:12.115608 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110199    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 20:38:12.115608 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110202    2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 20:38:12.115608 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110205    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 20:38:12.115996 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110207    2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 20:38:12.115996 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110210    2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 20:38:12.115996 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110214    2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 20:38:12.115996 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110218    2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 20:38:12.115996 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110221    2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 20:38:12.115996 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110224    2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 20:38:12.115996 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110232    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 20:38:12.115996 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110235    2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 20:38:12.115996 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110238    2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 20:38:12.115996 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110240    2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 20:38:12.115996 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110243    2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 20:38:12.115996 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110245    2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 20:38:12.115996 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110248    2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 20:38:12.115996 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110250    2583 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 20:38:12.115996 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110253    2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 20:38:12.115996 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110256    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 20:38:12.115996 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110258    2583 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 20:38:12.115996 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110261    2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 20:38:12.115996 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110263    2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 20:38:12.116453 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110265    2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 20:38:12.116453 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110268    2583 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 20:38:12.116453 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110271    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 20:38:12.116453 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110273    2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 20:38:12.116453 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110276    2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 20:38:12.116453 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110278    2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 20:38:12.116453 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110281    2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 20:38:12.116453 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110284    2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 20:38:12.116453 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110286    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 20:38:12.116453 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110288    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 20:38:12.116453 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110291    2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 20:38:12.116453 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110293    2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 20:38:12.116453 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110296    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 20:38:12.116453 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110298    2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 20:38:12.116453 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110301    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 20:38:12.116453 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110303    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 20:38:12.116453 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110306    2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 20:38:12.116453 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110308    2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 20:38:12.116453 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110311    2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 20:38:12.116453 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110313    2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 20:38:12.116911 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110335    2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 20:38:12.116911 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110339    2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 20:38:12.116911 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110342    2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 20:38:12.116911 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110345    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 20:38:12.116911 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110354    2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 20:38:12.116911 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110357    2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 20:38:12.116911 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110359    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 20:38:12.116911 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110362    2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 20:38:12.116911 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110364    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 20:38:12.116911 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110368    2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 20:38:12.116911 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110372    2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 20:38:12.116911 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110375    2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 20:38:12.116911 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110377    2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 20:38:12.116911 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110380    2583 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 20:38:12.116911 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110382    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 20:38:12.116911 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110385    2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 20:38:12.116911 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110387    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 20:38:12.116911 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110390    2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 20:38:12.116911 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110392    2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 20:38:12.117412 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110395    2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 20:38:12.117412 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110397    2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 20:38:12.117412 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110400    2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 20:38:12.117412 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110403    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 20:38:12.117412 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110405    2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 20:38:12.117412 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110408    2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 20:38:12.117412 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110411    2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 20:38:12.117412 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110414    2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 20:38:12.117412 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110417    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 20:38:12.117412 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110419    2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 20:38:12.117412 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110422    2583 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 20:38:12.117412 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110424    2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 20:38:12.117412 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110427    2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 20:38:12.117412 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110430    2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 20:38:12.117412 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110438    2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 20:38:12.117412 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110441    2583 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 20:38:12.117412 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110443    2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 20:38:12.117412 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110446    2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 20:38:12.117412 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110448    2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 20:38:12.117412 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110451    2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 20:38:12.117994 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110454    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 20:38:12.117994 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110456    2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 20:38:12.117994 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110459    2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 20:38:12.117994 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110461    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 20:38:12.117994 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110464    2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 20:38:12.117994 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110466    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 20:38:12.117994 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110468    2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 20:38:12.117994 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110471    2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 20:38:12.117994 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110473    2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 20:38:12.117994 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110476    2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 20:38:12.117994 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110478    2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 20:38:12.117994 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110481    2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 20:38:12.117994 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110484    2583 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 20:38:12.117994 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.110508    2583 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 20:38:12.117994 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110549    2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 20:38:12.117994 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110553    2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 20:38:12.118607 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110556    2583 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 20:38:12.118607 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110559    2583 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 20:38:12.118607 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110562    2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 20:38:12.118607 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110565    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 20:38:12.118607 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110567    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 20:38:12.118607 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110570    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 20:38:12.118607 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110573    2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 20:38:12.118607 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110575    2583 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 20:38:12.118607 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110578    2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 20:38:12.118607 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110580    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 20:38:12.118607 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110583    2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 20:38:12.118607 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110591    2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 20:38:12.118607 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110594    2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 20:38:12.118607 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110596    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 20:38:12.118607 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110599    2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 20:38:12.118607 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110601    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 20:38:12.118607 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110604    2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 20:38:12.118607 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110606    2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 20:38:12.118607 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110610    2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 20:38:12.119143 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110615    2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 20:38:12.119143 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110618    2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 20:38:12.119143 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110621    2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 20:38:12.119143 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110624    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 20:38:12.119143 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110626    2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 20:38:12.119143 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110629    2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 20:38:12.119143 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110632    2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 20:38:12.119143 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110634    2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 20:38:12.119143 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110637    2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 20:38:12.119143 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110639    2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 20:38:12.119143 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110642    2583 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 20:38:12.119143 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110644    2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 20:38:12.119143 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110647    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 20:38:12.119143 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110649    2583 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 20:38:12.119143 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110651    2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 20:38:12.119143 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110654    2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 20:38:12.119143 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110657    2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 20:38:12.119143 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110660    2583 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 20:38:12.119143 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110663    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 20:38:12.119143 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110666    2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 20:38:12.119638 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110668    2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 20:38:12.119638 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110670    2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 20:38:12.119638 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110673    2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 20:38:12.119638 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110675    2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 20:38:12.119638 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110678    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 20:38:12.119638 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110686    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 20:38:12.119638 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110688    2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 20:38:12.119638 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110691    2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 20:38:12.119638 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110693    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 20:38:12.119638 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110696    2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 20:38:12.119638 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110698    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 20:38:12.119638 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110700    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 20:38:12.119638 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110703    2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 20:38:12.119638 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110706    2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 20:38:12.119638 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110708    2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 20:38:12.119638 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110710    2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 20:38:12.119638 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110713    2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 20:38:12.119638 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110715    2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 20:38:12.119638 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110718    2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 20:38:12.119638 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110720    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 20:38:12.120171 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110722    2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 20:38:12.120171 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110725    2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 20:38:12.120171 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110727    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 20:38:12.120171 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110730    2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 20:38:12.120171 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110732    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 20:38:12.120171 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110736    2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 20:38:12.120171 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110740    2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 20:38:12.120171 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110743    2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 20:38:12.120171 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110746    2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 20:38:12.120171 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110749    2583 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 20:38:12.120171 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110751    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 20:38:12.120171 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110754    2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 20:38:12.120171 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110756    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 20:38:12.120171 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110759    2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 20:38:12.120171 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110761    2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 20:38:12.120171 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110764    2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 20:38:12.120171 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110766    2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 20:38:12.120171 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110769    2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 20:38:12.120171 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110777    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 20:38:12.120859 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110779    2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 20:38:12.120859 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110782    2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 20:38:12.120859 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110784    2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 20:38:12.120859 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110787    2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 20:38:12.120859 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110790    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 20:38:12.120859 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110793    2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 20:38:12.120859 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110795    2583 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 20:38:12.120859 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110798    2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 20:38:12.120859 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110801    2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 20:38:12.120859 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110803    2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 20:38:12.120859 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110806    2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 20:38:12.120859 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110808    2583 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 20:38:12.120859 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110811    2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 20:38:12.120859 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110813    2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 20:38:12.120859 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110816    2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 20:38:12.120859 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110818    2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 20:38:12.120859 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110821    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 20:38:12.120859 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110823    2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 20:38:12.120859 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110826    2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 20:38:12.120859 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110828    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 20:38:12.123507 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110831    2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 20:38:12.123507 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110833    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 20:38:12.123507 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110836    2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 20:38:12.123507 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110839    2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 20:38:12.123507 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110841    2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 20:38:12.123507 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:12.110843    2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 20:38:12.123507 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.110909    2583 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 20:38:12.123507 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.113591    2583 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Oct 30 20:38:12.123507 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.114511    2583 server.go:1014] "Starting client certificate rotation"
Oct 30 20:38:12.123507 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.114604    2583 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 20:38:12.123507 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.114639    2583 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 20:38:12.123507 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.116395    2583 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 20:38:12.123507 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.123364    2583 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 20:38:12.133388 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.133370    2583 log.go:25] "Validated CRI v1 runtime API"
Oct 30 20:38:12.136654 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.136641    2583 log.go:25] "Validated CRI v1 image API"
Oct 30 20:38:12.137511 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.137485    2583 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 20:38:12.140079 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.140061    2583 fs.go:135] Filesystem UUIDs: map[548b60ec-451a-4113-b9ab-28b655e56cec:/dev/nvme0n1p3 62db3ad3-a467-4a0c-9144-c7af3bc828a7:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
Oct 30 20:38:12.140137 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.140079    2583 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Oct 30 20:38:12.145384 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.145252    2583 manager.go:217] Machine: {Timestamp:2026-10-30 20:38:12.143825526 +0000 UTC m=+0.320508065 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100825 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2533b818d446c5b6c4b18c80a8918b SystemUUID:ec2533b8-18d4-46c5-b6c4-b18c80a8918b BootID:d8c270a2-8685-49ae-a7de-36fcea22edd0 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:f3:8d:bb:bf Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:f3:8d:bb:bf Speed:0 Mtu:9001} {Name:ovs-system MacAddress:be:7a:29:98:9a:ba Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 20:38:12.145967 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.145956    2583 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 20:38:12.146094 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.146083    2583 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 20:38:12.149591 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.149566    2583 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 20:38:12.149742 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.149593    2583 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-125.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 20:38:12.150244 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.150235    2583 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 20:38:12.150280 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.150246    2583 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 20:38:12.150280 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.150274    2583 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 20:38:12.150352 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.150289    2583 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 20:38:12.151163 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.151152    2583 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 20:38:12.153732 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.153714    2583 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 20:38:12.155413 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.155402    2583 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 20:38:12.155527 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.155518    2583 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 20:38:12.155555 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.155552    2583 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 20:38:12.155582 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.155563    2583 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 20:38:12.155582 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.155572    2583 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 20:38:12.155582 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.155582    2583 kubelet.go:405] "Adding apiserver pod source"
Oct 30 20:38:12.155684 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.155590    2583 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 20:38:12.155983 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.155966    2583 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 20:38:12.158259 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.158220    2583 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 20:38:12.164728 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.164703    2583 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 20:38:12.165604 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.165586    2583 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 20:38:12.165677 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.165632    2583 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 20:38:12.165819 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.165808    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 20:38:12.165852 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.165825    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 20:38:12.165852 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.165833    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 20:38:12.165852 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.165839    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 20:38:12.165852 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.165846    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 20:38:12.165981 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.165856    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 20:38:12.166504 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.166493    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 20:38:12.166720 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.166709    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 20:38:12.166759 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.166726    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 20:38:12.166759 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.166735    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 20:38:12.166759 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.166757    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 20:38:12.166859 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.166773    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 20:38:12.167640 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.167628    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 20:38:12.167640 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.167641    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 20:38:12.171642 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.171625    2583 server.go:1263] "Started kubelet"
Oct 30 20:38:12.171764 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.171702    2583 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 20:38:12.171850 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.171722    2583 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 20:38:12.171850 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.171802    2583 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 20:38:12.172545 ip-10-0-131-125 systemd[1]: Started Kubernetes Kubelet.
Oct 30 20:38:12.172905 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.172886    2583 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 20:38:12.173041 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.173019    2583 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 20:38:12.173391 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.173367    2583 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-131-125.us-east-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 30 20:38:12.176611 ip-10-0-131-125 kubenswrapper[2583]: E1030 20:38:12.176585    2583 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-131-125.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Oct 30 20:38:12.176681 ip-10-0-131-125 kubenswrapper[2583]: E1030 20:38:12.176585    2583 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Oct 30 20:38:12.176907 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.176784    2583 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-46cfh"
Oct 30 20:38:12.179001 ip-10-0-131-125 kubenswrapper[2583]: E1030 20:38:12.178972    2583 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Oct 30 20:38:12.180149 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.180133    2583 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 20:38:12.181193 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.181175    2583 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 20:38:12.181299 ip-10-0-131-125 kubenswrapper[2583]: E1030 20:38:12.181279    2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-125.us-east-2.compute.internal\" not found"
Oct 30 20:38:12.181558 ip-10-0-131-125 kubenswrapper[2583]: E1030 20:38:12.180417    2583 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-125.us-east-2.compute.internal.18e36926d7ab6909  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-125.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-131-125.us-east-2.compute.internal,},FirstTimestamp:2026-10-30 20:38:12.171598089 +0000 UTC m=+0.348280626,LastTimestamp:2026-10-30 20:38:12.171598089 +0000 UTC m=+0.348280626,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-125.us-east-2.compute.internal,}"
Oct 30 20:38:12.181810 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.181780    2583 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 20:38:12.181810 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.181783    2583 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 20:38:12.181942 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.181813    2583 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 20:38:12.181942 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.181933    2583 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 20:38:12.181942 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.181940    2583 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 20:38:12.182058 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.182038    2583 factory.go:55] Registering systemd factory
Oct 30 20:38:12.182058 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.182055    2583 factory.go:223] Registration of the systemd container factory successfully
Oct 30 20:38:12.182293 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.182280    2583 factory.go:153] Registering CRI-O factory
Oct 30 20:38:12.182353 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.182298    2583 factory.go:223] Registration of the crio container factory successfully
Oct 30 20:38:12.182353 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.182352    2583 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 20:38:12.182410 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.182366    2583 factory.go:103] Registering Raw factory
Oct 30 20:38:12.182410 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.182377    2583 manager.go:1199] Started watching for new ooms in manager
Oct 30 20:38:12.183359 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.183344    2583 manager.go:319] Starting recovery of all containers
Oct 30 20:38:12.187472 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.187455    2583 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-46cfh"
Oct 30 20:38:12.196848 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.196826    2583 manager.go:324] Recovery completed
Oct 30 20:38:12.197769 ip-10-0-131-125 kubenswrapper[2583]: E1030 20:38:12.197737    2583 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-131-125.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Oct 30 20:38:12.201514 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.201500    2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 20:38:12.203950 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.203933    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-125.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 20:38:12.204016 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.203967    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-125.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 20:38:12.204016 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.203982    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-125.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 20:38:12.204627 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.204611    2583 cpu_manager.go:226] "Starting" policy="none"
Oct 30 20:38:12.204627 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.204626    2583 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 20:38:12.204753 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.204645    2583 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 20:38:12.204753 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.204647    2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 20:38:12.205414 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.205400    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-125.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 20:38:12.205476 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.205421    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-125.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 20:38:12.205476 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.205438    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-125.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 20:38:12.206577 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.206565    2583 policy_none.go:50] "Start"
Oct 30 20:38:12.206625 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.206582    2583 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 20:38:12.206625 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.206593    2583 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 20:38:12.208425 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.208414    2583 policy_none.go:44] "Start"
Oct 30 20:38:12.217255 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.217234    2583 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 20:38:12.263243 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.242996    2583 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 20:38:12.263243 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.248772    2583 manager.go:341] "Starting Device Plugin manager"
Oct 30 20:38:12.263243 ip-10-0-131-125 kubenswrapper[2583]: E1030 20:38:12.249300    2583 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 20:38:12.263243 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.249314    2583 server.go:87] "Starting device plugin registration server"
Oct 30 20:38:12.263243 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.249614    2583 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 20:38:12.263243 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.249626    2583 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 20:38:12.263243 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.249749    2583 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 20:38:12.263243 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.249855    2583 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 20:38:12.263243 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.249994    2583 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 20:38:12.263243 ip-10-0-131-125 kubenswrapper[2583]: E1030 20:38:12.250619    2583 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 20:38:12.263243 ip-10-0-131-125 kubenswrapper[2583]: E1030 20:38:12.250669    2583 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-131-125.us-east-2.compute.internal\" not found"
Oct 30 20:38:12.263243 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.253240    2583 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 20:38:12.263243 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.254460    2583 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 20:38:12.263243 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.254482    2583 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 20:38:12.263243 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.254503    2583 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 20:38:12.263243 ip-10-0-131-125 kubenswrapper[2583]: E1030 20:38:12.254542    2583 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Oct 30 20:38:12.263243 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.258280    2583 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 20:38:12.350072 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.349951    2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 20:38:12.352041 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.352018    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-125.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 20:38:12.352170 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.352057    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-125.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 20:38:12.352170 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.352076    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-125.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 20:38:12.352170 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.352109    2583 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-131-125.us-east-2.compute.internal"
Oct 30 20:38:12.354635 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.354609    2583 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-125.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-131-125.us-east-2.compute.internal"]
Oct 30 20:38:12.354698 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.354672    2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 20:38:12.355458 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.355443    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-125.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 20:38:12.355546 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.355473    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-125.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 20:38:12.355546 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.355493    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-125.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 20:38:12.357776 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.357762    2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 20:38:12.357931 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.357917    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-125.us-east-2.compute.internal"
Oct 30 20:38:12.358004 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.357954    2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 20:38:12.358456 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.358438    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-125.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 20:38:12.358540 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.358467    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-125.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 20:38:12.358540 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.358481    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-125.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 20:38:12.358540 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.358532    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-125.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 20:38:12.358690 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.358554    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-125.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 20:38:12.358690 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.358569    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-125.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 20:38:12.360639 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.360624    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-125.us-east-2.compute.internal"
Oct 30 20:38:12.360720 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.360651    2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 20:38:12.360916 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.360889    2583 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-131-125.us-east-2.compute.internal"
Oct 30 20:38:12.360975 ip-10-0-131-125 kubenswrapper[2583]: E1030 20:38:12.360915    2583 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-131-125.us-east-2.compute.internal\": node \"ip-10-0-131-125.us-east-2.compute.internal\" not found"
Oct 30 20:38:12.361444 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.361430    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-125.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 20:38:12.361500 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.361451    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-125.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 20:38:12.361500 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.361460    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-125.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 20:38:12.382609 ip-10-0-131-125 kubenswrapper[2583]: E1030 20:38:12.382589    2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-125.us-east-2.compute.internal\" not found"
Oct 30 20:38:12.383624 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.383607    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/cbbd659d200641ef129d88b1e4575eca-config\") pod \"kube-apiserver-proxy-ip-10-0-131-125.us-east-2.compute.internal\" (UID: \"cbbd659d200641ef129d88b1e4575eca\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-125.us-east-2.compute.internal"
Oct 30 20:38:12.383671 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.383634    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/73fe153e2da84792f2cfb4a59468aa21-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-125.us-east-2.compute.internal\" (UID: \"73fe153e2da84792f2cfb4a59468aa21\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-125.us-east-2.compute.internal"
Oct 30 20:38:12.383671 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.383653    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/73fe153e2da84792f2cfb4a59468aa21-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-125.us-east-2.compute.internal\" (UID: \"73fe153e2da84792f2cfb4a59468aa21\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-125.us-east-2.compute.internal"
Oct 30 20:38:12.396184 ip-10-0-131-125 kubenswrapper[2583]: E1030 20:38:12.396167    2583 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-125.us-east-2.compute.internal\" not found" node="ip-10-0-131-125.us-east-2.compute.internal"
Oct 30 20:38:12.398382 ip-10-0-131-125 kubenswrapper[2583]: E1030 20:38:12.398367    2583 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-125.us-east-2.compute.internal\" not found" node="ip-10-0-131-125.us-east-2.compute.internal"
Oct 30 20:38:12.483438 ip-10-0-131-125 kubenswrapper[2583]: E1030 20:38:12.483405    2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-125.us-east-2.compute.internal\" not found"
Oct 30 20:38:12.484538 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.484519    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/73fe153e2da84792f2cfb4a59468aa21-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-125.us-east-2.compute.internal\" (UID: \"73fe153e2da84792f2cfb4a59468aa21\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-125.us-east-2.compute.internal"
Oct 30 20:38:12.484594 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.484550    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/cbbd659d200641ef129d88b1e4575eca-config\") pod \"kube-apiserver-proxy-ip-10-0-131-125.us-east-2.compute.internal\" (UID: \"cbbd659d200641ef129d88b1e4575eca\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-125.us-east-2.compute.internal"
Oct 30 20:38:12.484594 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.484570    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/73fe153e2da84792f2cfb4a59468aa21-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-125.us-east-2.compute.internal\" (UID: \"73fe153e2da84792f2cfb4a59468aa21\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-125.us-east-2.compute.internal"
Oct 30 20:38:12.484657 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.484599    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/cbbd659d200641ef129d88b1e4575eca-config\") pod \"kube-apiserver-proxy-ip-10-0-131-125.us-east-2.compute.internal\" (UID: \"cbbd659d200641ef129d88b1e4575eca\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-125.us-east-2.compute.internal"
Oct 30 20:38:12.484657 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.484602    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/73fe153e2da84792f2cfb4a59468aa21-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-125.us-east-2.compute.internal\" (UID: \"73fe153e2da84792f2cfb4a59468aa21\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-125.us-east-2.compute.internal"
Oct 30 20:38:12.484657 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.484631    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/73fe153e2da84792f2cfb4a59468aa21-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-125.us-east-2.compute.internal\" (UID: \"73fe153e2da84792f2cfb4a59468aa21\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-125.us-east-2.compute.internal"
Oct 30 20:38:12.583868 ip-10-0-131-125 kubenswrapper[2583]: E1030 20:38:12.583832    2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-125.us-east-2.compute.internal\" not found"
Oct 30 20:38:12.684560 ip-10-0-131-125 kubenswrapper[2583]: E1030 20:38:12.684480    2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-125.us-east-2.compute.internal\" not found"
Oct 30 20:38:12.699732 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.699712    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-125.us-east-2.compute.internal"
Oct 30 20:38:12.701883 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:12.701863    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-125.us-east-2.compute.internal"
Oct 30 20:38:12.785199 ip-10-0-131-125 kubenswrapper[2583]: E1030 20:38:12.785012    2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-125.us-east-2.compute.internal\" not found"
Oct 30 20:38:12.885460 ip-10-0-131-125 kubenswrapper[2583]: E1030 20:38:12.885430    2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-125.us-east-2.compute.internal\" not found"
Oct 30 20:38:12.985981 ip-10-0-131-125 kubenswrapper[2583]: E1030 20:38:12.985914    2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-125.us-east-2.compute.internal\" not found"
Oct 30 20:38:13.086630 ip-10-0-131-125 kubenswrapper[2583]: E1030 20:38:13.086588    2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-125.us-east-2.compute.internal\" not found"
Oct 30 20:38:13.114521 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:13.114487    2583 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Oct 30 20:38:13.115009 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:13.114645    2583 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 30 20:38:13.115009 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:13.114676    2583 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 30 20:38:13.187064 ip-10-0-131-125 kubenswrapper[2583]: E1030 20:38:13.187037    2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-125.us-east-2.compute.internal\" not found"
Oct 30 20:38:13.193020 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:13.192988    2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 20:33:12 +0000 UTC" deadline="2028-05-17 09:13:13.518597205 +0000 UTC"
Oct 30 20:38:13.193020 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:13.193016    2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13548h35m0.32558396s"
Oct 30 20:38:13.213035 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:13.213014    2583 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 20:38:13.281941 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:13.281870    2583 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-125.us-east-2.compute.internal"
Oct 30 20:38:13.294871 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:13.294849    2583 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 20:38:13.296903 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:13.296888    2583 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-125.us-east-2.compute.internal"
Oct 30 20:38:13.306766 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:13.306743    2583 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 20:38:13.352512 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:13.352469    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73fe153e2da84792f2cfb4a59468aa21.slice/crio-80a26258aed10d49cf54793b32a42a97bcfc09cf988ca045765d9d4d2d6d19ed WatchSource:0}: Error finding container 80a26258aed10d49cf54793b32a42a97bcfc09cf988ca045765d9d4d2d6d19ed: Status 404 returned error can't find the container with id 80a26258aed10d49cf54793b32a42a97bcfc09cf988ca045765d9d4d2d6d19ed
Oct 30 20:38:13.354556 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:13.354526    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbbd659d200641ef129d88b1e4575eca.slice/crio-c16277f3b4f3f62456078fbb6f9d926a1cdcf5849788f53a8bf3d1b1ff6d0028 WatchSource:0}: Error finding container c16277f3b4f3f62456078fbb6f9d926a1cdcf5849788f53a8bf3d1b1ff6d0028: Status 404 returned error can't find the container with id c16277f3b4f3f62456078fbb6f9d926a1cdcf5849788f53a8bf3d1b1ff6d0028
Oct 30 20:38:13.358309 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:13.358288    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 20:38:13.480563 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:13.480531    2583 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 20:38:14.157749 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.157712    2583 apiserver.go:52] "Watching apiserver"
Oct 30 20:38:14.166466 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.166423    2583 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 20:38:14.166870 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.166847    2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-operator/iptables-alerter-9p5zk","openshift-multus/multus-additional-cni-plugins-p5hfb","openshift-multus/multus-kxvqp","openshift-multus/network-metrics-daemon-5bww9","openshift-ovn-kubernetes/ovnkube-node-mt2kl","kube-system/global-pull-secret-syncer-xkfsj","kube-system/kube-apiserver-proxy-ip-10-0-131-125.us-east-2.compute.internal","openshift-cluster-node-tuning-operator/tuned-pllmn","openshift-dns/node-resolver-4kpcm","openshift-network-diagnostics/network-check-target-f4cch","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kxsjg","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-125.us-east-2.compute.internal","kube-system/konnectivity-agent-qcrpl","openshift-image-registry/node-ca-2kdx9"]
Oct 30 20:38:14.169480 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.169441    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f4cch"
Oct 30 20:38:14.169616 ip-10-0-131-125 kubenswrapper[2583]: E1030 20:38:14.169576    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f4cch" podUID="7957ff19-e0fc-42a7-967d-6e96e4a88094"
Oct 30 20:38:14.172016 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.171728    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-qcrpl"
Oct 30 20:38:14.174158 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.174137    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-kxvqp"
Oct 30 20:38:14.175948 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.175864    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 20:38:14.176124 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.176105    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 20:38:14.176240 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.176168    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-r2q8w\""
Oct 30 20:38:14.176669 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.176645    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5bww9"
Oct 30 20:38:14.176778 ip-10-0-131-125 kubenswrapper[2583]: E1030 20:38:14.176726    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5bww9" podUID="6fdde40e-3330-4b84-a470-6a7e6a536dce"
Oct 30 20:38:14.178185 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.178165    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-wzc24\""
Oct 30 20:38:14.178457 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.178437    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 20:38:14.178550 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.178471    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 20:38:14.178550 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.178527    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 20:38:14.178550 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.178541    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 20:38:14.179493 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.179474    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-mt2kl"
Oct 30 20:38:14.180705 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.180686    2583 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Oct 30 20:38:14.181633 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.181605    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xkfsj"
Oct 30 20:38:14.181747 ip-10-0-131-125 kubenswrapper[2583]: E1030 20:38:14.181688    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xkfsj" podUID="a03cdbf5-91a2-41eb-8c1e-3c0f40da2632"
Oct 30 20:38:14.184496 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.184164    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-4kpcm"
Oct 30 20:38:14.184496 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.184450    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-cctqn\""
Oct 30 20:38:14.187142 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.187125    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-2kdx9"
Oct 30 20:38:14.189410 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.189373    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-p5hfb"
Oct 30 20:38:14.193887 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.193863    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-9p5zk"
Oct 30 20:38:14.196542 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.196518    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kxsjg"
Oct 30 20:38:14.199114 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.199089    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-pllmn"
Oct 30 20:38:14.200182 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.200159    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d585cbd8-cb89-4718-9af4-716f64cc2598-konnectivity-ca\") pod \"konnectivity-agent-qcrpl\" (UID: \"d585cbd8-cb89-4718-9af4-716f64cc2598\") " pod="kube-system/konnectivity-agent-qcrpl"
Oct 30 20:38:14.200291 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.200197    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c3967ccc-990a-42f5-845e-cbeec5e18be0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mt2kl\" (UID: \"c3967ccc-990a-42f5-845e-cbeec5e18be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-mt2kl"
Oct 30 20:38:14.200291 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.200228    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c3967ccc-990a-42f5-845e-cbeec5e18be0-ovn-node-metrics-cert\") pod \"ovnkube-node-mt2kl\" (UID: \"c3967ccc-990a-42f5-845e-cbeec5e18be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-mt2kl"
Oct 30 20:38:14.200291 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.200248    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c6467dfc-7d9c-4ba8-ac2e-2ce19a74a0d4-hosts-file\") pod \"node-resolver-4kpcm\" (UID: \"c6467dfc-7d9c-4ba8-ac2e-2ce19a74a0d4\") " pod="openshift-dns/node-resolver-4kpcm"
Oct 30 20:38:14.200291 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.200265    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vxjc\" (UniqueName: \"kubernetes.io/projected/c6467dfc-7d9c-4ba8-ac2e-2ce19a74a0d4-kube-api-access-4vxjc\") pod \"node-resolver-4kpcm\" (UID: \"c6467dfc-7d9c-4ba8-ac2e-2ce19a74a0d4\") " pod="openshift-dns/node-resolver-4kpcm"
Oct 30 20:38:14.200291 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.200283    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ee639a0c-5f44-4091-a876-fe9109934f74-host-var-lib-cni-bin\") pod \"multus-kxvqp\" (UID: \"ee639a0c-5f44-4091-a876-fe9109934f74\") " pod="openshift-multus/multus-kxvqp"
Oct 30 20:38:14.200585 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.200301    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ee639a0c-5f44-4091-a876-fe9109934f74-host-var-lib-kubelet\") pod \"multus-kxvqp\" (UID: \"ee639a0c-5f44-4091-a876-fe9109934f74\") " pod="openshift-multus/multus-kxvqp"
Oct 30 20:38:14.200585 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.200324    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ee639a0c-5f44-4091-a876-fe9109934f74-host-run-multus-certs\") pod \"multus-kxvqp\" (UID: \"ee639a0c-5f44-4091-a876-fe9109934f74\") " pod="openshift-multus/multus-kxvqp"
Oct 30 20:38:14.200585 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.200362    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ee639a0c-5f44-4091-a876-fe9109934f74-etc-kubernetes\") pod \"multus-kxvqp\" (UID: \"ee639a0c-5f44-4091-a876-fe9109934f74\") " pod="openshift-multus/multus-kxvqp"
Oct 30 20:38:14.200585 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.200378    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c3967ccc-990a-42f5-845e-cbeec5e18be0-etc-openvswitch\") pod \"ovnkube-node-mt2kl\" (UID: \"c3967ccc-990a-42f5-845e-cbeec5e18be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-mt2kl"
Oct 30 20:38:14.200585 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.200400    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c3967ccc-990a-42f5-845e-cbeec5e18be0-systemd-units\") pod \"ovnkube-node-mt2kl\" (UID: \"c3967ccc-990a-42f5-845e-cbeec5e18be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-mt2kl"
Oct 30 20:38:14.200585 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.200421    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c3967ccc-990a-42f5-845e-cbeec5e18be0-host-slash\") pod \"ovnkube-node-mt2kl\" (UID: \"c3967ccc-990a-42f5-845e-cbeec5e18be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-mt2kl"
Oct 30 20:38:14.200585 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.200436    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a03cdbf5-91a2-41eb-8c1e-3c0f40da2632-dbus\") pod \"global-pull-secret-syncer-xkfsj\" (UID: \"a03cdbf5-91a2-41eb-8c1e-3c0f40da2632\") " pod="kube-system/global-pull-secret-syncer-xkfsj"
Oct 30 20:38:14.200585 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.200496    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a03cdbf5-91a2-41eb-8c1e-3c0f40da2632-original-pull-secret\") pod \"global-pull-secret-syncer-xkfsj\" (UID: \"a03cdbf5-91a2-41eb-8c1e-3c0f40da2632\") " pod="kube-system/global-pull-secret-syncer-xkfsj"
Oct 30 20:38:14.200585 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.200562    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ee639a0c-5f44-4091-a876-fe9109934f74-system-cni-dir\") pod \"multus-kxvqp\" (UID: \"ee639a0c-5f44-4091-a876-fe9109934f74\") " pod="openshift-multus/multus-kxvqp"
Oct 30 20:38:14.201052 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.200607    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ee639a0c-5f44-4091-a876-fe9109934f74-host-var-lib-cni-multus\") pod \"multus-kxvqp\" (UID: \"ee639a0c-5f44-4091-a876-fe9109934f74\") " pod="openshift-multus/multus-kxvqp"
Oct 30 20:38:14.201052 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.200633    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c3967ccc-990a-42f5-845e-cbeec5e18be0-ovnkube-config\") pod \"ovnkube-node-mt2kl\" (UID: \"c3967ccc-990a-42f5-845e-cbeec5e18be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-mt2kl"
Oct 30 20:38:14.201052 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.200679    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c3967ccc-990a-42f5-845e-cbeec5e18be0-host-run-netns\") pod \"ovnkube-node-mt2kl\" (UID: \"c3967ccc-990a-42f5-845e-cbeec5e18be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-mt2kl"
Oct 30 20:38:14.201052 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.200706    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d585cbd8-cb89-4718-9af4-716f64cc2598-agent-certs\") pod \"konnectivity-agent-qcrpl\" (UID: \"d585cbd8-cb89-4718-9af4-716f64cc2598\") " pod="kube-system/konnectivity-agent-qcrpl"
Oct 30 20:38:14.201052 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.200731    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ee639a0c-5f44-4091-a876-fe9109934f74-multus-cni-dir\") pod \"multus-kxvqp\" (UID: \"ee639a0c-5f44-4091-a876-fe9109934f74\") " pod="openshift-multus/multus-kxvqp"
Oct 30 20:38:14.201052 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.200755    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ee639a0c-5f44-4091-a876-fe9109934f74-host-run-k8s-cni-cncf-io\") pod \"multus-kxvqp\" (UID: \"ee639a0c-5f44-4091-a876-fe9109934f74\") " pod="openshift-multus/multus-kxvqp"
Oct 30 20:38:14.201052 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.200804    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ee639a0c-5f44-4091-a876-fe9109934f74-multus-conf-dir\") pod \"multus-kxvqp\" (UID: \"ee639a0c-5f44-4091-a876-fe9109934f74\") " pod="openshift-multus/multus-kxvqp"
Oct 30 20:38:14.201052 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.200820    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gftnn\" (UniqueName: \"kubernetes.io/projected/ee639a0c-5f44-4091-a876-fe9109934f74-kube-api-access-gftnn\") pod \"multus-kxvqp\" (UID: \"ee639a0c-5f44-4091-a876-fe9109934f74\") " pod="openshift-multus/multus-kxvqp"
Oct 30 20:38:14.201052 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.200858    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c3967ccc-990a-42f5-845e-cbeec5e18be0-host-kubelet\") pod \"ovnkube-node-mt2kl\" (UID: \"c3967ccc-990a-42f5-845e-cbeec5e18be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-mt2kl"
Oct 30 20:38:14.201052 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.200885    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c3967ccc-990a-42f5-845e-cbeec5e18be0-host-run-ovn-kubernetes\") pod \"ovnkube-node-mt2kl\" (UID: \"c3967ccc-990a-42f5-845e-cbeec5e18be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-mt2kl"
Oct 30 20:38:14.201052 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.200904    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ee639a0c-5f44-4091-a876-fe9109934f74-cni-binary-copy\") pod \"multus-kxvqp\" (UID: \"ee639a0c-5f44-4091-a876-fe9109934f74\") " pod="openshift-multus/multus-kxvqp"
Oct 30 20:38:14.201052 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.200919    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ee639a0c-5f44-4091-a876-fe9109934f74-host-run-netns\") pod \"multus-kxvqp\" (UID: \"ee639a0c-5f44-4091-a876-fe9109934f74\") " pod="openshift-multus/multus-kxvqp"
Oct 30 20:38:14.201052 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.200952    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ee639a0c-5f44-4091-a876-fe9109934f74-multus-daemon-config\") pod \"multus-kxvqp\" (UID: \"ee639a0c-5f44-4091-a876-fe9109934f74\") " pod="openshift-multus/multus-kxvqp"
Oct 30 20:38:14.201052 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.200977    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6fdde40e-3330-4b84-a470-6a7e6a536dce-metrics-certs\") pod \"network-metrics-daemon-5bww9\" (UID: \"6fdde40e-3330-4b84-a470-6a7e6a536dce\") " pod="openshift-multus/network-metrics-daemon-5bww9"
Oct 30 20:38:14.201052 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.201027    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5hb6\" (UniqueName: \"kubernetes.io/projected/6fdde40e-3330-4b84-a470-6a7e6a536dce-kube-api-access-p5hb6\") pod \"network-metrics-daemon-5bww9\" (UID: \"6fdde40e-3330-4b84-a470-6a7e6a536dce\") " pod="openshift-multus/network-metrics-daemon-5bww9"
Oct 30 20:38:14.201052 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.201053    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c3967ccc-990a-42f5-845e-cbeec5e18be0-var-lib-openvswitch\") pod \"ovnkube-node-mt2kl\" (UID: \"c3967ccc-990a-42f5-845e-cbeec5e18be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-mt2kl"
Oct 30 20:38:14.201863 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.201075    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c3967ccc-990a-42f5-845e-cbeec5e18be0-node-log\") pod \"ovnkube-node-mt2kl\" (UID: \"c3967ccc-990a-42f5-845e-cbeec5e18be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-mt2kl"
Oct 30 20:38:14.201863 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.201099    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c3967ccc-990a-42f5-845e-cbeec5e18be0-host-cni-netd\") pod \"ovnkube-node-mt2kl\" (UID: \"c3967ccc-990a-42f5-845e-cbeec5e18be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-mt2kl"
Oct 30 20:38:14.201863 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.201123    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ee639a0c-5f44-4091-a876-fe9109934f74-hostroot\") pod \"multus-kxvqp\" (UID: \"ee639a0c-5f44-4091-a876-fe9109934f74\") " pod="openshift-multus/multus-kxvqp"
Oct 30 20:38:14.201863 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.201471    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c3967ccc-990a-42f5-845e-cbeec5e18be0-run-openvswitch\") pod \"ovnkube-node-mt2kl\" (UID: \"c3967ccc-990a-42f5-845e-cbeec5e18be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-mt2kl"
Oct 30 20:38:14.201863 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.201502    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c3967ccc-990a-42f5-845e-cbeec5e18be0-run-ovn\") pod \"ovnkube-node-mt2kl\" (UID: \"c3967ccc-990a-42f5-845e-cbeec5e18be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-mt2kl"
Oct 30 20:38:14.201863 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.201528    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c3967ccc-990a-42f5-845e-cbeec5e18be0-log-socket\") pod \"ovnkube-node-mt2kl\" (UID: \"c3967ccc-990a-42f5-845e-cbeec5e18be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-mt2kl"
Oct 30 20:38:14.201863 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.201552    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c3967ccc-990a-42f5-845e-cbeec5e18be0-host-cni-bin\") pod \"ovnkube-node-mt2kl\" (UID: \"c3967ccc-990a-42f5-845e-cbeec5e18be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-mt2kl"
Oct 30 20:38:14.201863 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.201578    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c3967ccc-990a-42f5-845e-cbeec5e18be0-env-overrides\") pod \"ovnkube-node-mt2kl\" (UID: \"c3967ccc-990a-42f5-845e-cbeec5e18be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-mt2kl"
Oct 30 20:38:14.201863 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.201605    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6nk5\" (UniqueName: \"kubernetes.io/projected/c3967ccc-990a-42f5-845e-cbeec5e18be0-kube-api-access-z6nk5\") pod \"ovnkube-node-mt2kl\" (UID: \"c3967ccc-990a-42f5-845e-cbeec5e18be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-mt2kl"
Oct 30 20:38:14.201863 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.201641    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqttz\" (UniqueName: \"kubernetes.io/projected/7957ff19-e0fc-42a7-967d-6e96e4a88094-kube-api-access-pqttz\") pod \"network-check-target-f4cch\" (UID: \"7957ff19-e0fc-42a7-967d-6e96e4a88094\") " pod="openshift-network-diagnostics/network-check-target-f4cch"
Oct 30 20:38:14.201863 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.201669    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ee639a0c-5f44-4091-a876-fe9109934f74-cnibin\") pod \"multus-kxvqp\" (UID: \"ee639a0c-5f44-4091-a876-fe9109934f74\") " pod="openshift-multus/multus-kxvqp"
Oct 30 20:38:14.201863 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.201694    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ee639a0c-5f44-4091-a876-fe9109934f74-multus-socket-dir-parent\") pod \"multus-kxvqp\" (UID: \"ee639a0c-5f44-4091-a876-fe9109934f74\") " pod="openshift-multus/multus-kxvqp"
Oct 30 20:38:14.201863 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.201719    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c3967ccc-990a-42f5-845e-cbeec5e18be0-run-systemd\") pod \"ovnkube-node-mt2kl\" (UID: \"c3967ccc-990a-42f5-845e-cbeec5e18be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-mt2kl"
Oct 30 20:38:14.201863 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.201837    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c3967ccc-990a-42f5-845e-cbeec5e18be0-ovnkube-script-lib\") pod \"ovnkube-node-mt2kl\" (UID: \"c3967ccc-990a-42f5-845e-cbeec5e18be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-mt2kl"
Oct 30 20:38:14.201863 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.201865    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a03cdbf5-91a2-41eb-8c1e-3c0f40da2632-kubelet-config\") pod \"global-pull-secret-syncer-xkfsj\" (UID: \"a03cdbf5-91a2-41eb-8c1e-3c0f40da2632\") " pod="kube-system/global-pull-secret-syncer-xkfsj"
Oct 30 20:38:14.202630 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.201890    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6467dfc-7d9c-4ba8-ac2e-2ce19a74a0d4-tmp-dir\") pod \"node-resolver-4kpcm\" (UID: \"c6467dfc-7d9c-4ba8-ac2e-2ce19a74a0d4\") " pod="openshift-dns/node-resolver-4kpcm"
Oct 30 20:38:14.202630 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.202567    2583 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 20:38:14.235347 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.235318    2583 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-hmk8z"
Oct 30 20:38:14.253295 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.253268    2583 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-hmk8z"
Oct 30 20:38:14.261242 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.261189    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-125.us-east-2.compute.internal" event={"ID":"cbbd659d200641ef129d88b1e4575eca","Type":"ContainerStarted","Data":"c16277f3b4f3f62456078fbb6f9d926a1cdcf5849788f53a8bf3d1b1ff6d0028"}
Oct 30 20:38:14.263154 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.263130    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-125.us-east-2.compute.internal" event={"ID":"73fe153e2da84792f2cfb4a59468aa21","Type":"ContainerStarted","Data":"80a26258aed10d49cf54793b32a42a97bcfc09cf988ca045765d9d4d2d6d19ed"}
Oct 30 20:38:14.274885 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.274843    2583 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 20:38:14.283090 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.283059    2583 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 20:38:14.284876 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.284850    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 20:38:14.285062 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.284898    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 20:38:14.285164 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.285103    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 20:38:14.286203 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.286178    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 20:38:14.286321 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.286215    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-mnk4k\""
Oct 30 20:38:14.286425 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.286407    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 20:38:14.286510 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.286473    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 20:38:14.286601 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.286571    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 20:38:14.286676 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.286575    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-z6v2d\""
Oct 30 20:38:14.286733 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.286722    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 20:38:14.286781 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.286755    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 20:38:14.286781 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.286774    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-9r6zk\""
Oct 30 20:38:14.286888 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.286788    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 20:38:14.286888 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.286803    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 20:38:14.286888 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.286816    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 20:38:14.287027 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.286895    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 20:38:14.287027 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.286953    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-nxdl7\""
Oct 30 20:38:14.287194 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.287176    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 20:38:14.287194 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.287178    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 20:38:14.287194 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.287183    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 20:38:14.287404 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.287197    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 20:38:14.292904 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.292766    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 20:38:14.292904 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.292774    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 20:38:14.292904 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.292804    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 20:38:14.292904 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.292814    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 20:38:14.292904 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.292859    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-9s58x\""
Oct 30 20:38:14.292904 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.292864    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-jrnvk\""
Oct 30 20:38:14.302144 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.302108    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d5907106-812a-440a-b47d-f1889569d3b0-etc-systemd\") pod \"tuned-pllmn\" (UID: \"d5907106-812a-440a-b47d-f1889569d3b0\") " pod="openshift-cluster-node-tuning-operator/tuned-pllmn"
Oct 30 20:38:14.302144 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.302145    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d5907106-812a-440a-b47d-f1889569d3b0-etc-tuned\") pod \"tuned-pllmn\" (UID: \"d5907106-812a-440a-b47d-f1889569d3b0\") " pod="openshift-cluster-node-tuning-operator/tuned-pllmn"
Oct 30 20:38:14.302363 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.302172    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e584a74f-f82c-4a3f-81d6-0b93601980dd-cnibin\") pod \"multus-additional-cni-plugins-p5hfb\" (UID: \"e584a74f-f82c-4a3f-81d6-0b93601980dd\") " pod="openshift-multus/multus-additional-cni-plugins-p5hfb"
Oct 30 20:38:14.302363 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.302265    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e584a74f-f82c-4a3f-81d6-0b93601980dd-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-p5hfb\" (UID: \"e584a74f-f82c-4a3f-81d6-0b93601980dd\") " pod="openshift-multus/multus-additional-cni-plugins-p5hfb"
Oct 30 20:38:14.302363 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.302308    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f831b1bf-4a1f-48b7-8242-85c063825277-iptables-alerter-script\") pod \"iptables-alerter-9p5zk\" (UID: \"f831b1bf-4a1f-48b7-8242-85c063825277\") " pod="openshift-network-operator/iptables-alerter-9p5zk"
Oct 30 20:38:14.302495 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.302447    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a03cdbf5-91a2-41eb-8c1e-3c0f40da2632-kubelet-config\") pod \"global-pull-secret-syncer-xkfsj\" (UID: \"a03cdbf5-91a2-41eb-8c1e-3c0f40da2632\") " pod="kube-system/global-pull-secret-syncer-xkfsj"
Oct 30 20:38:14.302495 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.302477    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ee639a0c-5f44-4091-a876-fe9109934f74-multus-cni-dir\") pod \"multus-kxvqp\" (UID: \"ee639a0c-5f44-4091-a876-fe9109934f74\") " pod="openshift-multus/multus-kxvqp"
Oct 30 20:38:14.302600 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.302496    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ee639a0c-5f44-4091-a876-fe9109934f74-host-run-k8s-cni-cncf-io\") pod \"multus-kxvqp\" (UID: \"ee639a0c-5f44-4091-a876-fe9109934f74\") " pod="openshift-multus/multus-kxvqp"
Oct 30 20:38:14.302600 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.302522    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ee639a0c-5f44-4091-a876-fe9109934f74-multus-conf-dir\") pod \"multus-kxvqp\" (UID: \"ee639a0c-5f44-4091-a876-fe9109934f74\") " pod="openshift-multus/multus-kxvqp"
Oct 30 20:38:14.302600 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.302539    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ee639a0c-5f44-4091-a876-fe9109934f74-host-run-netns\") pod \"multus-kxvqp\" (UID: \"ee639a0c-5f44-4091-a876-fe9109934f74\") " pod="openshift-multus/multus-kxvqp"
Oct 30 20:38:14.302600 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.302555    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6fdde40e-3330-4b84-a470-6a7e6a536dce-metrics-certs\") pod \"network-metrics-daemon-5bww9\" (UID: \"6fdde40e-3330-4b84-a470-6a7e6a536dce\") " pod="openshift-multus/network-metrics-daemon-5bww9"
Oct 30 20:38:14.302600 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.302565    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a03cdbf5-91a2-41eb-8c1e-3c0f40da2632-kubelet-config\") pod \"global-pull-secret-syncer-xkfsj\" (UID: \"a03cdbf5-91a2-41eb-8c1e-3c0f40da2632\") " pod="kube-system/global-pull-secret-syncer-xkfsj"
Oct 30 20:38:14.302600 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.302593    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ee639a0c-5f44-4091-a876-fe9109934f74-host-run-k8s-cni-cncf-io\") pod \"multus-kxvqp\" (UID: \"ee639a0c-5f44-4091-a876-fe9109934f74\") " pod="openshift-multus/multus-kxvqp"
Oct 30 20:38:14.302912 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.302602    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ee639a0c-5f44-4091-a876-fe9109934f74-multus-conf-dir\") pod \"multus-kxvqp\" (UID: \"ee639a0c-5f44-4091-a876-fe9109934f74\") " pod="openshift-multus/multus-kxvqp"
Oct 30 20:38:14.302912 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.302618    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ee639a0c-5f44-4091-a876-fe9109934f74-host-run-netns\") pod \"multus-kxvqp\" (UID: \"ee639a0c-5f44-4091-a876-fe9109934f74\") " pod="openshift-multus/multus-kxvqp"
Oct 30 20:38:14.302912 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.302667    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5hb6\" (UniqueName: \"kubernetes.io/projected/6fdde40e-3330-4b84-a470-6a7e6a536dce-kube-api-access-p5hb6\") pod \"network-metrics-daemon-5bww9\" (UID: \"6fdde40e-3330-4b84-a470-6a7e6a536dce\") " pod="openshift-multus/network-metrics-daemon-5bww9"
Oct 30 20:38:14.302912 ip-10-0-131-125 kubenswrapper[2583]: E1030 20:38:14.302685    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 20:38:14.302912 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.302697    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c3967ccc-990a-42f5-845e-cbeec5e18be0-var-lib-openvswitch\") pod \"ovnkube-node-mt2kl\" (UID: \"c3967ccc-990a-42f5-845e-cbeec5e18be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-mt2kl"
Oct 30 20:38:14.302912 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.302729    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c3967ccc-990a-42f5-845e-cbeec5e18be0-node-log\") pod \"ovnkube-node-mt2kl\" (UID: \"c3967ccc-990a-42f5-845e-cbeec5e18be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-mt2kl"
Oct 30 20:38:14.302912 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.302757    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ee639a0c-5f44-4091-a876-fe9109934f74-multus-cni-dir\") pod \"multus-kxvqp\" (UID: \"ee639a0c-5f44-4091-a876-fe9109934f74\") " pod="openshift-multus/multus-kxvqp"
Oct 30 20:38:14.302912 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.302774    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c3967ccc-990a-42f5-845e-cbeec5e18be0-var-lib-openvswitch\") pod \"ovnkube-node-mt2kl\" (UID: \"c3967ccc-990a-42f5-845e-cbeec5e18be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-mt2kl"
Oct 30 20:38:14.302912 ip-10-0-131-125 kubenswrapper[2583]: E1030 20:38:14.302779    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6fdde40e-3330-4b84-a470-6a7e6a536dce-metrics-certs podName:6fdde40e-3330-4b84-a470-6a7e6a536dce nodeName:}" failed. No retries permitted until 2026-10-30 20:38:14.80273484 +0000 UTC m=+2.979417391 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6fdde40e-3330-4b84-a470-6a7e6a536dce-metrics-certs") pod "network-metrics-daemon-5bww9" (UID: "6fdde40e-3330-4b84-a470-6a7e6a536dce") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 20:38:14.302912 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.302823    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/76fb0a52-8bca-4610-9891-605df420cc7a-registration-dir\") pod \"aws-ebs-csi-driver-node-kxsjg\" (UID: \"76fb0a52-8bca-4610-9891-605df420cc7a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kxsjg"
Oct 30 20:38:14.302912 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.302847    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8flrx\" (UniqueName: \"kubernetes.io/projected/76fb0a52-8bca-4610-9891-605df420cc7a-kube-api-access-8flrx\") pod \"aws-ebs-csi-driver-node-kxsjg\" (UID: \"76fb0a52-8bca-4610-9891-605df420cc7a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kxsjg"
Oct 30 20:38:14.302912 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.302877    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c3967ccc-990a-42f5-845e-cbeec5e18be0-node-log\") pod \"ovnkube-node-mt2kl\" (UID: \"c3967ccc-990a-42f5-845e-cbeec5e18be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-mt2kl"
Oct 30 20:38:14.302912 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.302903    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d5907106-812a-440a-b47d-f1889569d3b0-host\") pod \"tuned-pllmn\" (UID: \"d5907106-812a-440a-b47d-f1889569d3b0\") " pod="openshift-cluster-node-tuning-operator/tuned-pllmn"
Oct 30 20:38:14.303609 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.302926    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ee639a0c-5f44-4091-a876-fe9109934f74-hostroot\") pod \"multus-kxvqp\" (UID: \"ee639a0c-5f44-4091-a876-fe9109934f74\") " pod="openshift-multus/multus-kxvqp"
Oct 30 20:38:14.303609 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.302965    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ee639a0c-5f44-4091-a876-fe9109934f74-hostroot\") pod \"multus-kxvqp\" (UID: \"ee639a0c-5f44-4091-a876-fe9109934f74\") " pod="openshift-multus/multus-kxvqp"
Oct 30 20:38:14.303609 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.302967    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c3967ccc-990a-42f5-845e-cbeec5e18be0-run-ovn\") pod \"ovnkube-node-mt2kl\" (UID: \"c3967ccc-990a-42f5-845e-cbeec5e18be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-mt2kl"
Oct 30 20:38:14.303609 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.303003    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c3967ccc-990a-42f5-845e-cbeec5e18be0-host-cni-bin\") pod \"ovnkube-node-mt2kl\" (UID: \"c3967ccc-990a-42f5-845e-cbeec5e18be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-mt2kl"
Oct 30 20:38:14.303609 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.303026    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f831b1bf-4a1f-48b7-8242-85c063825277-host-slash\") pod \"iptables-alerter-9p5zk\" (UID: \"f831b1bf-4a1f-48b7-8242-85c063825277\") " pod="openshift-network-operator/iptables-alerter-9p5zk"
Oct 30 20:38:14.303609 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.303026    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c3967ccc-990a-42f5-845e-cbeec5e18be0-run-ovn\") pod \"ovnkube-node-mt2kl\" (UID: \"c3967ccc-990a-42f5-845e-cbeec5e18be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-mt2kl"
Oct 30 20:38:14.303609 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.303047    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ee639a0c-5f44-4091-a876-fe9109934f74-cnibin\") pod \"multus-kxvqp\" (UID: \"ee639a0c-5f44-4091-a876-fe9109934f74\") " pod="openshift-multus/multus-kxvqp"
Oct 30 20:38:14.303609 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.303074    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ee639a0c-5f44-4091-a876-fe9109934f74-multus-socket-dir-parent\") pod \"multus-kxvqp\" (UID: \"ee639a0c-5f44-4091-a876-fe9109934f74\") " pod="openshift-multus/multus-kxvqp"
Oct 30 20:38:14.303609 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.303099    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c3967ccc-990a-42f5-845e-cbeec5e18be0-run-systemd\") pod \"ovnkube-node-mt2kl\" (UID: \"c3967ccc-990a-42f5-845e-cbeec5e18be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-mt2kl"
Oct 30 20:38:14.303609 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.303099    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c3967ccc-990a-42f5-845e-cbeec5e18be0-host-cni-bin\") pod \"ovnkube-node-mt2kl\" (UID: \"c3967ccc-990a-42f5-845e-cbeec5e18be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-mt2kl"
Oct 30 20:38:14.303609 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.303145    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c3967ccc-990a-42f5-845e-cbeec5e18be0-ovnkube-script-lib\") pod \"ovnkube-node-mt2kl\" (UID: \"c3967ccc-990a-42f5-845e-cbeec5e18be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-mt2kl"
Oct 30 20:38:14.303609 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.303215    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c3967ccc-990a-42f5-845e-cbeec5e18be0-run-systemd\") pod \"ovnkube-node-mt2kl\" (UID: \"c3967ccc-990a-42f5-845e-cbeec5e18be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-mt2kl"
Oct 30 20:38:14.303609 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.303239    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ee639a0c-5f44-4091-a876-fe9109934f74-multus-socket-dir-parent\") pod \"multus-kxvqp\" (UID: \"ee639a0c-5f44-4091-a876-fe9109934f74\") " pod="openshift-multus/multus-kxvqp"
Oct 30 20:38:14.303609 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.303250    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d5907106-812a-440a-b47d-f1889569d3b0-etc-sysctl-conf\") pod \"tuned-pllmn\" (UID: \"d5907106-812a-440a-b47d-f1889569d3b0\") " pod="openshift-cluster-node-tuning-operator/tuned-pllmn"
Oct 30 20:38:14.303609 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.303280    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c3967ccc-990a-42f5-845e-cbeec5e18be0-ovn-node-metrics-cert\") pod \"ovnkube-node-mt2kl\" (UID: \"c3967ccc-990a-42f5-845e-cbeec5e18be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-mt2kl"
Oct 30 20:38:14.303609 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.303290    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ee639a0c-5f44-4091-a876-fe9109934f74-cnibin\") pod \"multus-kxvqp\" (UID: \"ee639a0c-5f44-4091-a876-fe9109934f74\") " pod="openshift-multus/multus-kxvqp"
Oct 30 20:38:14.303609 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.303331    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e584a74f-f82c-4a3f-81d6-0b93601980dd-cni-binary-copy\") pod \"multus-additional-cni-plugins-p5hfb\" (UID: \"e584a74f-f82c-4a3f-81d6-0b93601980dd\") " pod="openshift-multus/multus-additional-cni-plugins-p5hfb"
Oct 30 20:38:14.303609 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.303376    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d5907106-812a-440a-b47d-f1889569d3b0-lib-modules\") pod \"tuned-pllmn\" (UID: \"d5907106-812a-440a-b47d-f1889569d3b0\") " pod="openshift-cluster-node-tuning-operator/tuned-pllmn"
Oct 30 20:38:14.304609 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.303399    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d5907106-812a-440a-b47d-f1889569d3b0-tmp\") pod \"tuned-pllmn\" (UID: \"d5907106-812a-440a-b47d-f1889569d3b0\") " pod="openshift-cluster-node-tuning-operator/tuned-pllmn"
Oct 30 20:38:14.304609 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.303833    2583 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 20:38:14.304609 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.304175    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ee639a0c-5f44-4091-a876-fe9109934f74-host-var-lib-kubelet\") pod \"multus-kxvqp\" (UID: \"ee639a0c-5f44-4091-a876-fe9109934f74\") " pod="openshift-multus/multus-kxvqp"
Oct 30 20:38:14.304609 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.304218    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ee639a0c-5f44-4091-a876-fe9109934f74-host-run-multus-certs\") pod \"multus-kxvqp\" (UID: \"ee639a0c-5f44-4091-a876-fe9109934f74\") " pod="openshift-multus/multus-kxvqp"
Oct 30 20:38:14.304609 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.304236    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c3967ccc-990a-42f5-845e-cbeec5e18be0-host-slash\") pod \"ovnkube-node-mt2kl\" (UID: \"c3967ccc-990a-42f5-845e-cbeec5e18be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-mt2kl"
Oct 30 20:38:14.304609 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.304269    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/880d0493-e39d-4d9c-b58b-aa48c33aeaef-serviceca\") pod \"node-ca-2kdx9\" (UID: \"880d0493-e39d-4d9c-b58b-aa48c33aeaef\") " pod="openshift-image-registry/node-ca-2kdx9"
Oct 30 20:38:14.304609 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.304293    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6h77j\" (UniqueName: \"kubernetes.io/projected/d5907106-812a-440a-b47d-f1889569d3b0-kube-api-access-6h77j\") pod \"tuned-pllmn\" (UID: \"d5907106-812a-440a-b47d-f1889569d3b0\") " pod="openshift-cluster-node-tuning-operator/tuned-pllmn"
Oct 30 20:38:14.304609 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.304317    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ee639a0c-5f44-4091-a876-fe9109934f74-system-cni-dir\") pod \"multus-kxvqp\" (UID: \"ee639a0c-5f44-4091-a876-fe9109934f74\") " pod="openshift-multus/multus-kxvqp"
Oct 30 20:38:14.304609 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.304356    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ee639a0c-5f44-4091-a876-fe9109934f74-host-var-lib-cni-multus\") pod \"multus-kxvqp\" (UID: \"ee639a0c-5f44-4091-a876-fe9109934f74\") " pod="openshift-multus/multus-kxvqp"
Oct 30 20:38:14.304609 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.304402    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c3967ccc-990a-42f5-845e-cbeec5e18be0-ovnkube-config\") pod \"ovnkube-node-mt2kl\" (UID: \"c3967ccc-990a-42f5-845e-cbeec5e18be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-mt2kl"
Oct 30 20:38:14.304609 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.304426    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d5907106-812a-440a-b47d-f1889569d3b0-etc-sysctl-d\") pod \"tuned-pllmn\" (UID: \"d5907106-812a-440a-b47d-f1889569d3b0\") " pod="openshift-cluster-node-tuning-operator/tuned-pllmn"
Oct 30 20:38:14.304609 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.304447    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d5907106-812a-440a-b47d-f1889569d3b0-run\") pod \"tuned-pllmn\" (UID: \"d5907106-812a-440a-b47d-f1889569d3b0\") " pod="openshift-cluster-node-tuning-operator/tuned-pllmn"
Oct 30 20:38:14.304609 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.304463    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6467dfc-7d9c-4ba8-ac2e-2ce19a74a0d4-tmp-dir\") pod \"node-resolver-4kpcm\" (UID: \"c6467dfc-7d9c-4ba8-ac2e-2ce19a74a0d4\") " pod="openshift-dns/node-resolver-4kpcm"
Oct 30 20:38:14.304609 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.304471    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ee639a0c-5f44-4091-a876-fe9109934f74-host-run-multus-certs\") pod \"multus-kxvqp\" (UID: \"ee639a0c-5f44-4091-a876-fe9109934f74\") " pod="openshift-multus/multus-kxvqp"
Oct 30 20:38:14.304609 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.304509    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c3967ccc-990a-42f5-845e-cbeec5e18be0-host-run-netns\") pod \"ovnkube-node-mt2kl\" (UID: \"c3967ccc-990a-42f5-845e-cbeec5e18be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-mt2kl"
Oct 30 20:38:14.304609 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.304532    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ee639a0c-5f44-4091-a876-fe9109934f74-host-var-lib-cni-multus\") pod \"multus-kxvqp\" (UID: \"ee639a0c-5f44-4091-a876-fe9109934f74\") " pod="openshift-multus/multus-kxvqp"
Oct 30 20:38:14.304609 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.304557    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c3967ccc-990a-42f5-845e-cbeec5e18be0-host-slash\") pod \"ovnkube-node-mt2kl\" (UID: \"c3967ccc-990a-42f5-845e-cbeec5e18be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-mt2kl"
Oct 30 20:38:14.304609 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.304565    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ee639a0c-5f44-4091-a876-fe9109934f74-system-cni-dir\") pod \"multus-kxvqp\" (UID: \"ee639a0c-5f44-4091-a876-fe9109934f74\") " pod="openshift-multus/multus-kxvqp"
Oct 30 20:38:14.305553 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.304578    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ee639a0c-5f44-4091-a876-fe9109934f74-host-var-lib-kubelet\") pod \"multus-kxvqp\" (UID: \"ee639a0c-5f44-4091-a876-fe9109934f74\") " pod="openshift-multus/multus-kxvqp"
Oct 30 20:38:14.305553 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.304478    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c3967ccc-990a-42f5-845e-cbeec5e18be0-host-run-netns\") pod \"ovnkube-node-mt2kl\" (UID: \"c3967ccc-990a-42f5-845e-cbeec5e18be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-mt2kl"
Oct 30 20:38:14.305553 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.304623    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d585cbd8-cb89-4718-9af4-716f64cc2598-agent-certs\") pod \"konnectivity-agent-qcrpl\" (UID: \"d585cbd8-cb89-4718-9af4-716f64cc2598\") " pod="kube-system/konnectivity-agent-qcrpl"
Oct 30 20:38:14.305553 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.304676    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gftnn\" (UniqueName: \"kubernetes.io/projected/ee639a0c-5f44-4091-a876-fe9109934f74-kube-api-access-gftnn\") pod \"multus-kxvqp\" (UID: \"ee639a0c-5f44-4091-a876-fe9109934f74\") " pod="openshift-multus/multus-kxvqp"
Oct 30 20:38:14.305553 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.304703    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c3967ccc-990a-42f5-845e-cbeec5e18be0-host-kubelet\") pod \"ovnkube-node-mt2kl\" (UID: \"c3967ccc-990a-42f5-845e-cbeec5e18be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-mt2kl"
Oct 30 20:38:14.305553 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.304730    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c3967ccc-990a-42f5-845e-cbeec5e18be0-host-run-ovn-kubernetes\") pod \"ovnkube-node-mt2kl\" (UID: \"c3967ccc-990a-42f5-845e-cbeec5e18be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-mt2kl"
Oct 30 20:38:14.305553 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.304734    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c3967ccc-990a-42f5-845e-cbeec5e18be0-host-kubelet\") pod \"ovnkube-node-mt2kl\" (UID: \"c3967ccc-990a-42f5-845e-cbeec5e18be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-mt2kl"
Oct 30 20:38:14.305553 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.304761    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnt7j\" (UniqueName: \"kubernetes.io/projected/f831b1bf-4a1f-48b7-8242-85c063825277-kube-api-access-mnt7j\") pod \"iptables-alerter-9p5zk\" (UID: \"f831b1bf-4a1f-48b7-8242-85c063825277\") " pod="openshift-network-operator/iptables-alerter-9p5zk"
Oct 30 20:38:14.305553 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.304882    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c3967ccc-990a-42f5-845e-cbeec5e18be0-host-run-ovn-kubernetes\") pod \"ovnkube-node-mt2kl\" (UID: \"c3967ccc-990a-42f5-845e-cbeec5e18be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-mt2kl"
Oct 30 20:38:14.305553 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.304912    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d5907106-812a-440a-b47d-f1889569d3b0-etc-kubernetes\") pod \"tuned-pllmn\" (UID: \"d5907106-812a-440a-b47d-f1889569d3b0\") " pod="openshift-cluster-node-tuning-operator/tuned-pllmn"
Oct 30 20:38:14.305553 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.304956    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ee639a0c-5f44-4091-a876-fe9109934f74-cni-binary-copy\") pod \"multus-kxvqp\" (UID: \"ee639a0c-5f44-4091-a876-fe9109934f74\") " pod="openshift-multus/multus-kxvqp"
Oct 30 20:38:14.305553 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.304991    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ee639a0c-5f44-4091-a876-fe9109934f74-multus-daemon-config\") pod \"multus-kxvqp\" (UID: \"ee639a0c-5f44-4091-a876-fe9109934f74\") " pod="openshift-multus/multus-kxvqp"
Oct 30 20:38:14.305553 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.305017    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c3967ccc-990a-42f5-845e-cbeec5e18be0-host-cni-netd\") pod \"ovnkube-node-mt2kl\" (UID: \"c3967ccc-990a-42f5-845e-cbeec5e18be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-mt2kl"
Oct 30 20:38:14.305553 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.305043    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c3967ccc-990a-42f5-845e-cbeec5e18be0-env-overrides\") pod \"ovnkube-node-mt2kl\" (UID: \"c3967ccc-990a-42f5-845e-cbeec5e18be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-mt2kl"
Oct 30 20:38:14.305553 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.305076    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vxjc\" (UniqueName: \"kubernetes.io/projected/c6467dfc-7d9c-4ba8-ac2e-2ce19a74a0d4-kube-api-access-4vxjc\") pod \"node-resolver-4kpcm\" (UID: \"c6467dfc-7d9c-4ba8-ac2e-2ce19a74a0d4\") " pod="openshift-dns/node-resolver-4kpcm"
Oct 30 20:38:14.305553 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.305093    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d5907106-812a-440a-b47d-f1889569d3b0-etc-sysconfig\") pod \"tuned-pllmn\" (UID: \"d5907106-812a-440a-b47d-f1889569d3b0\") " pod="openshift-cluster-node-tuning-operator/tuned-pllmn"
Oct 30 20:38:14.305553 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.305116    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d5907106-812a-440a-b47d-f1889569d3b0-sys\") pod \"tuned-pllmn\" (UID: \"d5907106-812a-440a-b47d-f1889569d3b0\") " pod="openshift-cluster-node-tuning-operator/tuned-pllmn"
Oct 30 20:38:14.306413 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.305142    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d5907106-812a-440a-b47d-f1889569d3b0-var-lib-kubelet\") pod \"tuned-pllmn\" (UID: \"d5907106-812a-440a-b47d-f1889569d3b0\") " pod="openshift-cluster-node-tuning-operator/tuned-pllmn"
Oct 30 20:38:14.306413 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.305185    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c3967ccc-990a-42f5-845e-cbeec5e18be0-run-openvswitch\") pod \"ovnkube-node-mt2kl\" (UID: \"c3967ccc-990a-42f5-845e-cbeec5e18be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-mt2kl"
Oct 30 20:38:14.306413 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.305203    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c3967ccc-990a-42f5-845e-cbeec5e18be0-log-socket\") pod \"ovnkube-node-mt2kl\" (UID: \"c3967ccc-990a-42f5-845e-cbeec5e18be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-mt2kl"
Oct 30 20:38:14.306413 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.305230    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6nk5\" (UniqueName: \"kubernetes.io/projected/c3967ccc-990a-42f5-845e-cbeec5e18be0-kube-api-access-z6nk5\") pod \"ovnkube-node-mt2kl\" (UID: \"c3967ccc-990a-42f5-845e-cbeec5e18be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-mt2kl"
Oct 30 20:38:14.306413 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.305254    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/76fb0a52-8bca-4610-9891-605df420cc7a-etc-selinux\") pod \"aws-ebs-csi-driver-node-kxsjg\" (UID: \"76fb0a52-8bca-4610-9891-605df420cc7a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kxsjg"
Oct 30 20:38:14.306413 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.305282    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d5907106-812a-440a-b47d-f1889569d3b0-etc-modprobe-d\") pod \"tuned-pllmn\" (UID: \"d5907106-812a-440a-b47d-f1889569d3b0\") " pod="openshift-cluster-node-tuning-operator/tuned-pllmn"
Oct 30 20:38:14.306413 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.305309    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqttz\" (UniqueName: \"kubernetes.io/projected/7957ff19-e0fc-42a7-967d-6e96e4a88094-kube-api-access-pqttz\") pod \"network-check-target-f4cch\" (UID: \"7957ff19-e0fc-42a7-967d-6e96e4a88094\") " pod="openshift-network-diagnostics/network-check-target-f4cch"
Oct 30 20:38:14.306413 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.305335    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fj44\" (UniqueName: \"kubernetes.io/projected/880d0493-e39d-4d9c-b58b-aa48c33aeaef-kube-api-access-9fj44\") pod \"node-ca-2kdx9\" (UID: \"880d0493-e39d-4d9c-b58b-aa48c33aeaef\") " pod="openshift-image-registry/node-ca-2kdx9"
Oct 30 20:38:14.306413 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.305411    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e584a74f-f82c-4a3f-81d6-0b93601980dd-tuning-conf-dir\") pod \"multus-additional-cni-plugins-p5hfb\" (UID: \"e584a74f-f82c-4a3f-81d6-0b93601980dd\") " pod="openshift-multus/multus-additional-cni-plugins-p5hfb"
Oct 30 20:38:14.306413 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.305437    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d585cbd8-cb89-4718-9af4-716f64cc2598-konnectivity-ca\") pod \"konnectivity-agent-qcrpl\" (UID: \"d585cbd8-cb89-4718-9af4-716f64cc2598\") " pod="kube-system/konnectivity-agent-qcrpl"
Oct 30 20:38:14.306413 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.305439    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c3967ccc-990a-42f5-845e-cbeec5e18be0-ovnkube-script-lib\") pod \"ovnkube-node-mt2kl\" (UID: \"c3967ccc-990a-42f5-845e-cbeec5e18be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-mt2kl"
Oct 30 20:38:14.306413 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.305456    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c3967ccc-990a-42f5-845e-cbeec5e18be0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mt2kl\" (UID: \"c3967ccc-990a-42f5-845e-cbeec5e18be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-mt2kl"
Oct 30 20:38:14.306413 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.305483    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c6467dfc-7d9c-4ba8-ac2e-2ce19a74a0d4-hosts-file\") pod \"node-resolver-4kpcm\" (UID: \"c6467dfc-7d9c-4ba8-ac2e-2ce19a74a0d4\") " pod="openshift-dns/node-resolver-4kpcm"
Oct 30 20:38:14.306413 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.305508    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/880d0493-e39d-4d9c-b58b-aa48c33aeaef-host\") pod \"node-ca-2kdx9\" (UID: \"880d0493-e39d-4d9c-b58b-aa48c33aeaef\") " pod="openshift-image-registry/node-ca-2kdx9"
Oct 30 20:38:14.306413 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.305530    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e584a74f-f82c-4a3f-81d6-0b93601980dd-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-p5hfb\" (UID: \"e584a74f-f82c-4a3f-81d6-0b93601980dd\") " pod="openshift-multus/multus-additional-cni-plugins-p5hfb"
Oct 30 20:38:14.306413 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.305533    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c3967ccc-990a-42f5-845e-cbeec5e18be0-run-openvswitch\") pod \"ovnkube-node-mt2kl\" (UID: \"c3967ccc-990a-42f5-845e-cbeec5e18be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-mt2kl"
Oct 30 20:38:14.307205 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.305548    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smcsz\" (UniqueName: \"kubernetes.io/projected/e584a74f-f82c-4a3f-81d6-0b93601980dd-kube-api-access-smcsz\") pod \"multus-additional-cni-plugins-p5hfb\" (UID: \"e584a74f-f82c-4a3f-81d6-0b93601980dd\") " pod="openshift-multus/multus-additional-cni-plugins-p5hfb"
Oct 30 20:38:14.307205 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.305567    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/76fb0a52-8bca-4610-9891-605df420cc7a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kxsjg\" (UID: \"76fb0a52-8bca-4610-9891-605df420cc7a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kxsjg"
Oct 30 20:38:14.307205 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.305560    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c3967ccc-990a-42f5-845e-cbeec5e18be0-ovnkube-config\") pod \"ovnkube-node-mt2kl\" (UID: \"c3967ccc-990a-42f5-845e-cbeec5e18be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-mt2kl"
Oct 30 20:38:14.307205 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.305586    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ee639a0c-5f44-4091-a876-fe9109934f74-host-var-lib-cni-bin\") pod \"multus-kxvqp\" (UID: \"ee639a0c-5f44-4091-a876-fe9109934f74\") " pod="openshift-multus/multus-kxvqp"
Oct 30 20:38:14.307205 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.305670    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ee639a0c-5f44-4091-a876-fe9109934f74-etc-kubernetes\") pod \"multus-kxvqp\" (UID: \"ee639a0c-5f44-4091-a876-fe9109934f74\") " pod="openshift-multus/multus-kxvqp"
Oct 30 20:38:14.307205 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.305698    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c3967ccc-990a-42f5-845e-cbeec5e18be0-etc-openvswitch\") pod \"ovnkube-node-mt2kl\" (UID: \"c3967ccc-990a-42f5-845e-cbeec5e18be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-mt2kl"
Oct 30 20:38:14.307205 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.305721    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e584a74f-f82c-4a3f-81d6-0b93601980dd-system-cni-dir\") pod \"multus-additional-cni-plugins-p5hfb\" (UID: \"e584a74f-f82c-4a3f-81d6-0b93601980dd\") " pod="openshift-multus/multus-additional-cni-plugins-p5hfb"
Oct 30 20:38:14.307205 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.305740    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/76fb0a52-8bca-4610-9891-605df420cc7a-socket-dir\") pod \"aws-ebs-csi-driver-node-kxsjg\" (UID: \"76fb0a52-8bca-4610-9891-605df420cc7a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kxsjg"
Oct 30 20:38:14.307205 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.305756    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/76fb0a52-8bca-4610-9891-605df420cc7a-device-dir\") pod \"aws-ebs-csi-driver-node-kxsjg\" (UID: \"76fb0a52-8bca-4610-9891-605df420cc7a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kxsjg"
Oct 30 20:38:14.307205 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.305781    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c3967ccc-990a-42f5-845e-cbeec5e18be0-systemd-units\") pod \"ovnkube-node-mt2kl\" (UID: \"c3967ccc-990a-42f5-845e-cbeec5e18be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-mt2kl"
Oct 30 20:38:14.307205 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.305806    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a03cdbf5-91a2-41eb-8c1e-3c0f40da2632-dbus\") pod \"global-pull-secret-syncer-xkfsj\" (UID: \"a03cdbf5-91a2-41eb-8c1e-3c0f40da2632\") " pod="kube-system/global-pull-secret-syncer-xkfsj"
Oct 30 20:38:14.307205 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.305831    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a03cdbf5-91a2-41eb-8c1e-3c0f40da2632-original-pull-secret\") pod \"global-pull-secret-syncer-xkfsj\" (UID: \"a03cdbf5-91a2-41eb-8c1e-3c0f40da2632\") " pod="kube-system/global-pull-secret-syncer-xkfsj"
Oct 30 20:38:14.307205 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.305849    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/76fb0a52-8bca-4610-9891-605df420cc7a-sys-fs\") pod \"aws-ebs-csi-driver-node-kxsjg\" (UID: \"76fb0a52-8bca-4610-9891-605df420cc7a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kxsjg"
Oct 30 20:38:14.307205 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.305922    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c3967ccc-990a-42f5-845e-cbeec5e18be0-env-overrides\") pod \"ovnkube-node-mt2kl\" (UID: \"c3967ccc-990a-42f5-845e-cbeec5e18be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-mt2kl"
Oct 30 20:38:14.307205 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.305923    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ee639a0c-5f44-4091-a876-fe9109934f74-host-var-lib-cni-bin\") pod \"multus-kxvqp\" (UID: \"ee639a0c-5f44-4091-a876-fe9109934f74\") " pod="openshift-multus/multus-kxvqp"
Oct 30 20:38:14.307205 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.305952    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c3967ccc-990a-42f5-845e-cbeec5e18be0-log-socket\") pod \"ovnkube-node-mt2kl\" (UID: \"c3967ccc-990a-42f5-845e-cbeec5e18be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-mt2kl"
Oct 30 20:38:14.307205 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.306130    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c3967ccc-990a-42f5-845e-cbeec5e18be0-host-cni-netd\") pod \"ovnkube-node-mt2kl\" (UID: \"c3967ccc-990a-42f5-845e-cbeec5e18be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-mt2kl"
Oct 30 20:38:14.308108 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.306178    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ee639a0c-5f44-4091-a876-fe9109934f74-etc-kubernetes\") pod \"multus-kxvqp\" (UID: \"ee639a0c-5f44-4091-a876-fe9109934f74\") " pod="openshift-multus/multus-kxvqp"
Oct 30 20:38:14.308108 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.306224    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c3967ccc-990a-42f5-845e-cbeec5e18be0-systemd-units\") pod \"ovnkube-node-mt2kl\" (UID: \"c3967ccc-990a-42f5-845e-cbeec5e18be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-mt2kl"
Oct 30 20:38:14.308108 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.306225    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c3967ccc-990a-42f5-845e-cbeec5e18be0-etc-openvswitch\") pod \"ovnkube-node-mt2kl\" (UID: \"c3967ccc-990a-42f5-845e-cbeec5e18be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-mt2kl"
Oct 30 20:38:14.308108 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.306273    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a03cdbf5-91a2-41eb-8c1e-3c0f40da2632-dbus\") pod \"global-pull-secret-syncer-xkfsj\" (UID: \"a03cdbf5-91a2-41eb-8c1e-3c0f40da2632\") " pod="kube-system/global-pull-secret-syncer-xkfsj"
Oct 30 20:38:14.308108 ip-10-0-131-125 kubenswrapper[2583]: E1030 20:38:14.306440    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 20:38:14.308108 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.306467    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c3967ccc-990a-42f5-845e-cbeec5e18be0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mt2kl\" (UID: \"c3967ccc-990a-42f5-845e-cbeec5e18be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-mt2kl"
Oct 30 20:38:14.308108 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.306488    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c6467dfc-7d9c-4ba8-ac2e-2ce19a74a0d4-hosts-file\") pod \"node-resolver-4kpcm\" (UID: \"c6467dfc-7d9c-4ba8-ac2e-2ce19a74a0d4\") " pod="openshift-dns/node-resolver-4kpcm"
Oct 30 20:38:14.308108 ip-10-0-131-125 kubenswrapper[2583]: E1030 20:38:14.306495    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a03cdbf5-91a2-41eb-8c1e-3c0f40da2632-original-pull-secret podName:a03cdbf5-91a2-41eb-8c1e-3c0f40da2632 nodeName:}" failed. No retries permitted until 2026-10-30 20:38:14.806476988 +0000 UTC m=+2.983159532 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a03cdbf5-91a2-41eb-8c1e-3c0f40da2632-original-pull-secret") pod "global-pull-secret-syncer-xkfsj" (UID: "a03cdbf5-91a2-41eb-8c1e-3c0f40da2632") : object "kube-system"/"original-pull-secret" not registered
Oct 30 20:38:14.308108 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.306777    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6467dfc-7d9c-4ba8-ac2e-2ce19a74a0d4-tmp-dir\") pod \"node-resolver-4kpcm\" (UID: \"c6467dfc-7d9c-4ba8-ac2e-2ce19a74a0d4\") " pod="openshift-dns/node-resolver-4kpcm"
Oct 30 20:38:14.308108 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.307393    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d585cbd8-cb89-4718-9af4-716f64cc2598-konnectivity-ca\") pod \"konnectivity-agent-qcrpl\" (UID: \"d585cbd8-cb89-4718-9af4-716f64cc2598\") " pod="kube-system/konnectivity-agent-qcrpl"
Oct 30 20:38:14.308108 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.307497    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ee639a0c-5f44-4091-a876-fe9109934f74-multus-daemon-config\") pod \"multus-kxvqp\" (UID: \"ee639a0c-5f44-4091-a876-fe9109934f74\") " pod="openshift-multus/multus-kxvqp"
Oct 30 20:38:14.309401 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.309375    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ee639a0c-5f44-4091-a876-fe9109934f74-cni-binary-copy\") pod \"multus-kxvqp\" (UID: \"ee639a0c-5f44-4091-a876-fe9109934f74\") " pod="openshift-multus/multus-kxvqp"
Oct 30 20:38:14.310558 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.310535    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c3967ccc-990a-42f5-845e-cbeec5e18be0-ovn-node-metrics-cert\") pod \"ovnkube-node-mt2kl\" (UID: \"c3967ccc-990a-42f5-845e-cbeec5e18be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-mt2kl"
Oct 30 20:38:14.310727 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.310709    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d585cbd8-cb89-4718-9af4-716f64cc2598-agent-certs\") pod \"konnectivity-agent-qcrpl\" (UID: \"d585cbd8-cb89-4718-9af4-716f64cc2598\") " pod="kube-system/konnectivity-agent-qcrpl"
Oct 30 20:38:14.317681 ip-10-0-131-125 kubenswrapper[2583]: E1030 20:38:14.317656    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 20:38:14.317681 ip-10-0-131-125 kubenswrapper[2583]: E1030 20:38:14.317682    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 20:38:14.317835 ip-10-0-131-125 kubenswrapper[2583]: E1030 20:38:14.317694    2583 projected.go:196] Error preparing data for projected volume kube-api-access-pqttz for pod openshift-network-diagnostics/network-check-target-f4cch: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 20:38:14.317835 ip-10-0-131-125 kubenswrapper[2583]: E1030 20:38:14.317749    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7957ff19-e0fc-42a7-967d-6e96e4a88094-kube-api-access-pqttz podName:7957ff19-e0fc-42a7-967d-6e96e4a88094 nodeName:}" failed. No retries permitted until 2026-10-30 20:38:14.817733632 +0000 UTC m=+2.994416173 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-pqttz" (UniqueName: "kubernetes.io/projected/7957ff19-e0fc-42a7-967d-6e96e4a88094-kube-api-access-pqttz") pod "network-check-target-f4cch" (UID: "7957ff19-e0fc-42a7-967d-6e96e4a88094") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 20:38:14.318922 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.318895    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5hb6\" (UniqueName: \"kubernetes.io/projected/6fdde40e-3330-4b84-a470-6a7e6a536dce-kube-api-access-p5hb6\") pod \"network-metrics-daemon-5bww9\" (UID: \"6fdde40e-3330-4b84-a470-6a7e6a536dce\") " pod="openshift-multus/network-metrics-daemon-5bww9"
Oct 30 20:38:14.319650 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.319624    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gftnn\" (UniqueName: \"kubernetes.io/projected/ee639a0c-5f44-4091-a876-fe9109934f74-kube-api-access-gftnn\") pod \"multus-kxvqp\" (UID: \"ee639a0c-5f44-4091-a876-fe9109934f74\") " pod="openshift-multus/multus-kxvqp"
Oct 30 20:38:14.320153 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.320127    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vxjc\" (UniqueName: \"kubernetes.io/projected/c6467dfc-7d9c-4ba8-ac2e-2ce19a74a0d4-kube-api-access-4vxjc\") pod \"node-resolver-4kpcm\" (UID: \"c6467dfc-7d9c-4ba8-ac2e-2ce19a74a0d4\") " pod="openshift-dns/node-resolver-4kpcm"
Oct 30 20:38:14.321776 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.321744    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6nk5\" (UniqueName: \"kubernetes.io/projected/c3967ccc-990a-42f5-845e-cbeec5e18be0-kube-api-access-z6nk5\") pod \"ovnkube-node-mt2kl\" (UID: \"c3967ccc-990a-42f5-845e-cbeec5e18be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-mt2kl"
Oct 30 20:38:14.406850 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.406813    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f831b1bf-4a1f-48b7-8242-85c063825277-iptables-alerter-script\") pod \"iptables-alerter-9p5zk\" (UID: \"f831b1bf-4a1f-48b7-8242-85c063825277\") " pod="openshift-network-operator/iptables-alerter-9p5zk"
Oct 30 20:38:14.407019 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.406878    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/76fb0a52-8bca-4610-9891-605df420cc7a-registration-dir\") pod \"aws-ebs-csi-driver-node-kxsjg\" (UID: \"76fb0a52-8bca-4610-9891-605df420cc7a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kxsjg"
Oct 30 20:38:14.407019 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.406913    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8flrx\" (UniqueName: \"kubernetes.io/projected/76fb0a52-8bca-4610-9891-605df420cc7a-kube-api-access-8flrx\") pod \"aws-ebs-csi-driver-node-kxsjg\" (UID: \"76fb0a52-8bca-4610-9891-605df420cc7a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kxsjg"
Oct 30 20:38:14.407019 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.406943    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d5907106-812a-440a-b47d-f1889569d3b0-host\") pod \"tuned-pllmn\" (UID: \"d5907106-812a-440a-b47d-f1889569d3b0\") " pod="openshift-cluster-node-tuning-operator/tuned-pllmn"
Oct 30 20:38:14.407019 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.406973    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f831b1bf-4a1f-48b7-8242-85c063825277-host-slash\") pod \"iptables-alerter-9p5zk\" (UID: \"f831b1bf-4a1f-48b7-8242-85c063825277\") " pod="openshift-network-operator/iptables-alerter-9p5zk"
Oct 30 20:38:14.407019 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.406996    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d5907106-812a-440a-b47d-f1889569d3b0-etc-sysctl-conf\") pod \"tuned-pllmn\" (UID: \"d5907106-812a-440a-b47d-f1889569d3b0\") " pod="openshift-cluster-node-tuning-operator/tuned-pllmn"
Oct 30 20:38:14.407019 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.407018    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e584a74f-f82c-4a3f-81d6-0b93601980dd-cni-binary-copy\") pod \"multus-additional-cni-plugins-p5hfb\" (UID: \"e584a74f-f82c-4a3f-81d6-0b93601980dd\") " pod="openshift-multus/multus-additional-cni-plugins-p5hfb"
Oct 30 20:38:14.407357 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.407029    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/76fb0a52-8bca-4610-9891-605df420cc7a-registration-dir\") pod \"aws-ebs-csi-driver-node-kxsjg\" (UID: \"76fb0a52-8bca-4610-9891-605df420cc7a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kxsjg"
Oct 30 20:38:14.407357 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.407045    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d5907106-812a-440a-b47d-f1889569d3b0-lib-modules\") pod \"tuned-pllmn\" (UID: \"d5907106-812a-440a-b47d-f1889569d3b0\") " pod="openshift-cluster-node-tuning-operator/tuned-pllmn"
Oct 30 20:38:14.407357 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.407100    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f831b1bf-4a1f-48b7-8242-85c063825277-host-slash\") pod \"iptables-alerter-9p5zk\" (UID: \"f831b1bf-4a1f-48b7-8242-85c063825277\") " pod="openshift-network-operator/iptables-alerter-9p5zk"
Oct 30 20:38:14.407357 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.407155    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d5907106-812a-440a-b47d-f1889569d3b0-tmp\") pod \"tuned-pllmn\" (UID: \"d5907106-812a-440a-b47d-f1889569d3b0\") " pod="openshift-cluster-node-tuning-operator/tuned-pllmn"
Oct 30 20:38:14.407357 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.407192    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d5907106-812a-440a-b47d-f1889569d3b0-lib-modules\") pod \"tuned-pllmn\" (UID: \"d5907106-812a-440a-b47d-f1889569d3b0\") " pod="openshift-cluster-node-tuning-operator/tuned-pllmn"
Oct 30 20:38:14.407357 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.407191    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/880d0493-e39d-4d9c-b58b-aa48c33aeaef-serviceca\") pod \"node-ca-2kdx9\" (UID: \"880d0493-e39d-4d9c-b58b-aa48c33aeaef\") " pod="openshift-image-registry/node-ca-2kdx9"
Oct 30 20:38:14.407357 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.407207    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d5907106-812a-440a-b47d-f1889569d3b0-etc-sysctl-conf\") pod \"tuned-pllmn\" (UID: \"d5907106-812a-440a-b47d-f1889569d3b0\") " pod="openshift-cluster-node-tuning-operator/tuned-pllmn"
Oct 30 20:38:14.407357 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.407229    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6h77j\" (UniqueName: \"kubernetes.io/projected/d5907106-812a-440a-b47d-f1889569d3b0-kube-api-access-6h77j\") pod \"tuned-pllmn\" (UID: \"d5907106-812a-440a-b47d-f1889569d3b0\") " pod="openshift-cluster-node-tuning-operator/tuned-pllmn"
Oct 30 20:38:14.407357 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.407115    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d5907106-812a-440a-b47d-f1889569d3b0-host\") pod \"tuned-pllmn\" (UID: \"d5907106-812a-440a-b47d-f1889569d3b0\") " pod="openshift-cluster-node-tuning-operator/tuned-pllmn"
Oct 30 20:38:14.407357 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.407250    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d5907106-812a-440a-b47d-f1889569d3b0-etc-sysctl-d\") pod \"tuned-pllmn\" (UID: \"d5907106-812a-440a-b47d-f1889569d3b0\") " pod="openshift-cluster-node-tuning-operator/tuned-pllmn"
Oct 30 20:38:14.407357 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.407267    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d5907106-812a-440a-b47d-f1889569d3b0-run\") pod \"tuned-pllmn\" (UID: \"d5907106-812a-440a-b47d-f1889569d3b0\") " pod="openshift-cluster-node-tuning-operator/tuned-pllmn"
Oct 30 20:38:14.407357 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.407288    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnt7j\" (UniqueName: \"kubernetes.io/projected/f831b1bf-4a1f-48b7-8242-85c063825277-kube-api-access-mnt7j\") pod \"iptables-alerter-9p5zk\" (UID: \"f831b1bf-4a1f-48b7-8242-85c063825277\") " pod="openshift-network-operator/iptables-alerter-9p5zk"
Oct 30 20:38:14.407357 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.407305    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d5907106-812a-440a-b47d-f1889569d3b0-etc-kubernetes\") pod \"tuned-pllmn\" (UID: \"d5907106-812a-440a-b47d-f1889569d3b0\") " pod="openshift-cluster-node-tuning-operator/tuned-pllmn"
Oct 30 20:38:14.407357 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.407325    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d5907106-812a-440a-b47d-f1889569d3b0-etc-sysconfig\") pod \"tuned-pllmn\" (UID: \"d5907106-812a-440a-b47d-f1889569d3b0\") " pod="openshift-cluster-node-tuning-operator/tuned-pllmn"
Oct 30 20:38:14.407357 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.407342    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d5907106-812a-440a-b47d-f1889569d3b0-sys\") pod \"tuned-pllmn\" (UID: \"d5907106-812a-440a-b47d-f1889569d3b0\") " pod="openshift-cluster-node-tuning-operator/tuned-pllmn"
Oct 30 20:38:14.407357 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.407357    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d5907106-812a-440a-b47d-f1889569d3b0-var-lib-kubelet\") pod \"tuned-pllmn\" (UID: \"d5907106-812a-440a-b47d-f1889569d3b0\") " pod="openshift-cluster-node-tuning-operator/tuned-pllmn"
Oct 30 20:38:14.408105 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.407401    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/76fb0a52-8bca-4610-9891-605df420cc7a-etc-selinux\") pod \"aws-ebs-csi-driver-node-kxsjg\" (UID: \"76fb0a52-8bca-4610-9891-605df420cc7a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kxsjg"
Oct 30 20:38:14.408105 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.407425    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d5907106-812a-440a-b47d-f1889569d3b0-etc-modprobe-d\") pod \"tuned-pllmn\" (UID: \"d5907106-812a-440a-b47d-f1889569d3b0\") " pod="openshift-cluster-node-tuning-operator/tuned-pllmn"
Oct 30 20:38:14.408105 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.407462    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fj44\" (UniqueName: \"kubernetes.io/projected/880d0493-e39d-4d9c-b58b-aa48c33aeaef-kube-api-access-9fj44\") pod \"node-ca-2kdx9\" (UID: \"880d0493-e39d-4d9c-b58b-aa48c33aeaef\") " pod="openshift-image-registry/node-ca-2kdx9"
Oct 30 20:38:14.408105 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.407486    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e584a74f-f82c-4a3f-81d6-0b93601980dd-tuning-conf-dir\") pod \"multus-additional-cni-plugins-p5hfb\" (UID: \"e584a74f-f82c-4a3f-81d6-0b93601980dd\") " pod="openshift-multus/multus-additional-cni-plugins-p5hfb"
Oct 30 20:38:14.408105 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.407514    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/880d0493-e39d-4d9c-b58b-aa48c33aeaef-host\") pod \"node-ca-2kdx9\" (UID: \"880d0493-e39d-4d9c-b58b-aa48c33aeaef\") " pod="openshift-image-registry/node-ca-2kdx9"
Oct 30 20:38:14.408105 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.407524    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d5907106-812a-440a-b47d-f1889569d3b0-etc-sysconfig\") pod \"tuned-pllmn\" (UID: \"d5907106-812a-440a-b47d-f1889569d3b0\") " pod="openshift-cluster-node-tuning-operator/tuned-pllmn"
Oct 30 20:38:14.408105 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.407542    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e584a74f-f82c-4a3f-81d6-0b93601980dd-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-p5hfb\" (UID: \"e584a74f-f82c-4a3f-81d6-0b93601980dd\") " pod="openshift-multus/multus-additional-cni-plugins-p5hfb"
Oct 30 20:38:14.408105 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.407572    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smcsz\" (UniqueName: \"kubernetes.io/projected/e584a74f-f82c-4a3f-81d6-0b93601980dd-kube-api-access-smcsz\") pod \"multus-additional-cni-plugins-p5hfb\" (UID: \"e584a74f-f82c-4a3f-81d6-0b93601980dd\") " pod="openshift-multus/multus-additional-cni-plugins-p5hfb"
Oct 30 20:38:14.408105 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.407645    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f831b1bf-4a1f-48b7-8242-85c063825277-iptables-alerter-script\") pod \"iptables-alerter-9p5zk\" (UID: \"f831b1bf-4a1f-48b7-8242-85c063825277\") " pod="openshift-network-operator/iptables-alerter-9p5zk"
Oct 30 20:38:14.408105 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.407663    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d5907106-812a-440a-b47d-f1889569d3b0-etc-sysctl-d\") pod \"tuned-pllmn\" (UID: \"d5907106-812a-440a-b47d-f1889569d3b0\") " pod="openshift-cluster-node-tuning-operator/tuned-pllmn"
Oct 30 20:38:14.408105 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.407680    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d5907106-812a-440a-b47d-f1889569d3b0-etc-modprobe-d\") pod \"tuned-pllmn\" (UID: \"d5907106-812a-440a-b47d-f1889569d3b0\") " pod="openshift-cluster-node-tuning-operator/tuned-pllmn"
Oct 30 20:38:14.408105 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.407724    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d5907106-812a-440a-b47d-f1889569d3b0-etc-kubernetes\") pod \"tuned-pllmn\" (UID: \"d5907106-812a-440a-b47d-f1889569d3b0\") " pod="openshift-cluster-node-tuning-operator/tuned-pllmn"
Oct 30 20:38:14.408105 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.407723    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/880d0493-e39d-4d9c-b58b-aa48c33aeaef-serviceca\") pod \"node-ca-2kdx9\" (UID: \"880d0493-e39d-4d9c-b58b-aa48c33aeaef\") " pod="openshift-image-registry/node-ca-2kdx9"
Oct 30 20:38:14.408105 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.407726    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/76fb0a52-8bca-4610-9891-605df420cc7a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kxsjg\" (UID: \"76fb0a52-8bca-4610-9891-605df420cc7a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kxsjg"
Oct 30 20:38:14.408105 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.407778    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/76fb0a52-8bca-4610-9891-605df420cc7a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kxsjg\" (UID: \"76fb0a52-8bca-4610-9891-605df420cc7a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kxsjg"
Oct 30 20:38:14.408105 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.407781    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d5907106-812a-440a-b47d-f1889569d3b0-sys\") pod \"tuned-pllmn\" (UID: \"d5907106-812a-440a-b47d-f1889569d3b0\") " pod="openshift-cluster-node-tuning-operator/tuned-pllmn"
Oct 30 20:38:14.408105 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.407795    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e584a74f-f82c-4a3f-81d6-0b93601980dd-system-cni-dir\") pod \"multus-additional-cni-plugins-p5hfb\" (UID: \"e584a74f-f82c-4a3f-81d6-0b93601980dd\") " pod="openshift-multus/multus-additional-cni-plugins-p5hfb"
Oct 30 20:38:14.408732 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.407800    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e584a74f-f82c-4a3f-81d6-0b93601980dd-cni-binary-copy\") pod \"multus-additional-cni-plugins-p5hfb\" (UID: \"e584a74f-f82c-4a3f-81d6-0b93601980dd\") " pod="openshift-multus/multus-additional-cni-plugins-p5hfb"
Oct 30 20:38:14.408732 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.407824    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d5907106-812a-440a-b47d-f1889569d3b0-run\") pod \"tuned-pllmn\" (UID: \"d5907106-812a-440a-b47d-f1889569d3b0\") " pod="openshift-cluster-node-tuning-operator/tuned-pllmn"
Oct 30 20:38:14.408732 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.407825    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d5907106-812a-440a-b47d-f1889569d3b0-var-lib-kubelet\") pod \"tuned-pllmn\" (UID: \"d5907106-812a-440a-b47d-f1889569d3b0\") " pod="openshift-cluster-node-tuning-operator/tuned-pllmn"
Oct 30 20:38:14.408732 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.407824    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/76fb0a52-8bca-4610-9891-605df420cc7a-socket-dir\") pod \"aws-ebs-csi-driver-node-kxsjg\" (UID: \"76fb0a52-8bca-4610-9891-605df420cc7a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kxsjg"
Oct 30 20:38:14.408732 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.407870    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e584a74f-f82c-4a3f-81d6-0b93601980dd-system-cni-dir\") pod \"multus-additional-cni-plugins-p5hfb\" (UID: \"e584a74f-f82c-4a3f-81d6-0b93601980dd\") " pod="openshift-multus/multus-additional-cni-plugins-p5hfb"
Oct 30 20:38:14.408732 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.407874    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/880d0493-e39d-4d9c-b58b-aa48c33aeaef-host\") pod \"node-ca-2kdx9\" (UID: \"880d0493-e39d-4d9c-b58b-aa48c33aeaef\") " pod="openshift-image-registry/node-ca-2kdx9"
Oct 30 20:38:14.408732 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.407874    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/76fb0a52-8bca-4610-9891-605df420cc7a-device-dir\") pod \"aws-ebs-csi-driver-node-kxsjg\" (UID: \"76fb0a52-8bca-4610-9891-605df420cc7a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kxsjg"
Oct 30 20:38:14.408732 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.407919    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/76fb0a52-8bca-4610-9891-605df420cc7a-device-dir\") pod \"aws-ebs-csi-driver-node-kxsjg\" (UID: \"76fb0a52-8bca-4610-9891-605df420cc7a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kxsjg"
Oct 30 20:38:14.408732 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.407928    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/76fb0a52-8bca-4610-9891-605df420cc7a-etc-selinux\") pod \"aws-ebs-csi-driver-node-kxsjg\" (UID: \"76fb0a52-8bca-4610-9891-605df420cc7a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kxsjg"
Oct 30 20:38:14.408732 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.407927    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/76fb0a52-8bca-4610-9891-605df420cc7a-sys-fs\") pod \"aws-ebs-csi-driver-node-kxsjg\" (UID: \"76fb0a52-8bca-4610-9891-605df420cc7a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kxsjg"
Oct 30 20:38:14.408732 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.407965    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d5907106-812a-440a-b47d-f1889569d3b0-etc-systemd\") pod \"tuned-pllmn\" (UID: \"d5907106-812a-440a-b47d-f1889569d3b0\") " pod="openshift-cluster-node-tuning-operator/tuned-pllmn"
Oct 30 20:38:14.408732 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.407990    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d5907106-812a-440a-b47d-f1889569d3b0-etc-tuned\") pod \"tuned-pllmn\" (UID: \"d5907106-812a-440a-b47d-f1889569d3b0\") " pod="openshift-cluster-node-tuning-operator/tuned-pllmn"
Oct 30 20:38:14.408732 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.408018    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e584a74f-f82c-4a3f-81d6-0b93601980dd-cnibin\") pod \"multus-additional-cni-plugins-p5hfb\" (UID: \"e584a74f-f82c-4a3f-81d6-0b93601980dd\") " pod="openshift-multus/multus-additional-cni-plugins-p5hfb"
Oct 30 20:38:14.408732 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.408023    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e584a74f-f82c-4a3f-81d6-0b93601980dd-tuning-conf-dir\") pod \"multus-additional-cni-plugins-p5hfb\" (UID: \"e584a74f-f82c-4a3f-81d6-0b93601980dd\") " pod="openshift-multus/multus-additional-cni-plugins-p5hfb"
Oct 30 20:38:14.408732 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.408043    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d5907106-812a-440a-b47d-f1889569d3b0-etc-systemd\") pod \"tuned-pllmn\" (UID: \"d5907106-812a-440a-b47d-f1889569d3b0\") " pod="openshift-cluster-node-tuning-operator/tuned-pllmn"
Oct 30 20:38:14.408732 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.408045    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e584a74f-f82c-4a3f-81d6-0b93601980dd-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-p5hfb\" (UID: \"e584a74f-f82c-4a3f-81d6-0b93601980dd\") " pod="openshift-multus/multus-additional-cni-plugins-p5hfb"
Oct 30 20:38:14.408732 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.408044    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/76fb0a52-8bca-4610-9891-605df420cc7a-socket-dir\") pod \"aws-ebs-csi-driver-node-kxsjg\" (UID: \"76fb0a52-8bca-4610-9891-605df420cc7a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kxsjg"
Oct 30 20:38:14.409358 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.408084    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/76fb0a52-8bca-4610-9891-605df420cc7a-sys-fs\") pod \"aws-ebs-csi-driver-node-kxsjg\" (UID: \"76fb0a52-8bca-4610-9891-605df420cc7a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kxsjg"
Oct 30 20:38:14.409358 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.408123    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e584a74f-f82c-4a3f-81d6-0b93601980dd-cnibin\") pod \"multus-additional-cni-plugins-p5hfb\" (UID: \"e584a74f-f82c-4a3f-81d6-0b93601980dd\") " pod="openshift-multus/multus-additional-cni-plugins-p5hfb"
Oct 30 20:38:14.409358 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.408366    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e584a74f-f82c-4a3f-81d6-0b93601980dd-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-p5hfb\" (UID: \"e584a74f-f82c-4a3f-81d6-0b93601980dd\") " pod="openshift-multus/multus-additional-cni-plugins-p5hfb"
Oct 30 20:38:14.409358 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.408610    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e584a74f-f82c-4a3f-81d6-0b93601980dd-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-p5hfb\" (UID: \"e584a74f-f82c-4a3f-81d6-0b93601980dd\") " pod="openshift-multus/multus-additional-cni-plugins-p5hfb"
Oct 30 20:38:14.411209 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.411184    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d5907106-812a-440a-b47d-f1889569d3b0-etc-tuned\") pod \"tuned-pllmn\" (UID: \"d5907106-812a-440a-b47d-f1889569d3b0\") " pod="openshift-cluster-node-tuning-operator/tuned-pllmn"
Oct 30 20:38:14.411360 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.411315    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d5907106-812a-440a-b47d-f1889569d3b0-tmp\") pod \"tuned-pllmn\" (UID: \"d5907106-812a-440a-b47d-f1889569d3b0\") " pod="openshift-cluster-node-tuning-operator/tuned-pllmn"
Oct 30 20:38:14.420841 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.420784    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnt7j\" (UniqueName: \"kubernetes.io/projected/f831b1bf-4a1f-48b7-8242-85c063825277-kube-api-access-mnt7j\") pod \"iptables-alerter-9p5zk\" (UID: \"f831b1bf-4a1f-48b7-8242-85c063825277\") " pod="openshift-network-operator/iptables-alerter-9p5zk"
Oct 30 20:38:14.421684 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.421657    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6h77j\" (UniqueName: \"kubernetes.io/projected/d5907106-812a-440a-b47d-f1889569d3b0-kube-api-access-6h77j\") pod \"tuned-pllmn\" (UID: \"d5907106-812a-440a-b47d-f1889569d3b0\") " pod="openshift-cluster-node-tuning-operator/tuned-pllmn"
Oct 30 20:38:14.422319 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.422277    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fj44\" (UniqueName: \"kubernetes.io/projected/880d0493-e39d-4d9c-b58b-aa48c33aeaef-kube-api-access-9fj44\") pod \"node-ca-2kdx9\" (UID: \"880d0493-e39d-4d9c-b58b-aa48c33aeaef\") " pod="openshift-image-registry/node-ca-2kdx9"
Oct 30 20:38:14.422445 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.422399    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8flrx\" (UniqueName: \"kubernetes.io/projected/76fb0a52-8bca-4610-9891-605df420cc7a-kube-api-access-8flrx\") pod \"aws-ebs-csi-driver-node-kxsjg\" (UID: \"76fb0a52-8bca-4610-9891-605df420cc7a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kxsjg"
Oct 30 20:38:14.424149 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.424127    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smcsz\" (UniqueName: \"kubernetes.io/projected/e584a74f-f82c-4a3f-81d6-0b93601980dd-kube-api-access-smcsz\") pod \"multus-additional-cni-plugins-p5hfb\" (UID: \"e584a74f-f82c-4a3f-81d6-0b93601980dd\") " pod="openshift-multus/multus-additional-cni-plugins-p5hfb"
Oct 30 20:38:14.489190 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.489146    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-qcrpl"
Oct 30 20:38:14.502302 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:14.502271    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd585cbd8_cb89_4718_9af4_716f64cc2598.slice/crio-97605bd7a36f6cc05063a503084c058056dfa45b02a518afbf5402517fbc5e60 WatchSource:0}: Error finding container 97605bd7a36f6cc05063a503084c058056dfa45b02a518afbf5402517fbc5e60: Status 404 returned error can't find the container with id 97605bd7a36f6cc05063a503084c058056dfa45b02a518afbf5402517fbc5e60
Oct 30 20:38:14.503274 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.503249    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-kxvqp"
Oct 30 20:38:14.513438 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:14.512984    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee639a0c_5f44_4091_a876_fe9109934f74.slice/crio-0bcc74b6507dc0209f85ad2a19b8912b241efe513eb5ad35491b1737ca8a262c WatchSource:0}: Error finding container 0bcc74b6507dc0209f85ad2a19b8912b241efe513eb5ad35491b1737ca8a262c: Status 404 returned error can't find the container with id 0bcc74b6507dc0209f85ad2a19b8912b241efe513eb5ad35491b1737ca8a262c
Oct 30 20:38:14.529131 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.529103    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-4kpcm"
Oct 30 20:38:14.530491 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.530465    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-2kdx9"
Oct 30 20:38:14.535352 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.535147    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-mt2kl"
Oct 30 20:38:14.544427 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:14.544395    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6467dfc_7d9c_4ba8_ac2e_2ce19a74a0d4.slice/crio-d8c56fe4c6222b749344a9775e4bb3134b1453511b9b5929dc9ea72f76dbe863 WatchSource:0}: Error finding container d8c56fe4c6222b749344a9775e4bb3134b1453511b9b5929dc9ea72f76dbe863: Status 404 returned error can't find the container with id d8c56fe4c6222b749344a9775e4bb3134b1453511b9b5929dc9ea72f76dbe863
Oct 30 20:38:14.545474 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:14.545436    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod880d0493_e39d_4d9c_b58b_aa48c33aeaef.slice/crio-228acd337e25401e4710ece12fd7e71c0d4581a209aef19e35790ee08ea50a0e WatchSource:0}: Error finding container 228acd337e25401e4710ece12fd7e71c0d4581a209aef19e35790ee08ea50a0e: Status 404 returned error can't find the container with id 228acd337e25401e4710ece12fd7e71c0d4581a209aef19e35790ee08ea50a0e
Oct 30 20:38:14.548197 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:14.548142    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3967ccc_990a_42f5_845e_cbeec5e18be0.slice/crio-56461b767faa0395c16b73aa678e90856e30180670d31d772bd981dd7a51ceb5 WatchSource:0}: Error finding container 56461b767faa0395c16b73aa678e90856e30180670d31d772bd981dd7a51ceb5: Status 404 returned error can't find the container with id 56461b767faa0395c16b73aa678e90856e30180670d31d772bd981dd7a51ceb5
Oct 30 20:38:14.551051 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.551028    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-9p5zk"
Oct 30 20:38:14.559823 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.559617    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-p5hfb"
Oct 30 20:38:14.560782 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.560760    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kxsjg"
Oct 30 20:38:14.562449 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:14.562417    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf831b1bf_4a1f_48b7_8242_85c063825277.slice/crio-2d300712e711b03830289edf5cb38a7ecef41dad9f2269956fd7e6b3d7ada3b2 WatchSource:0}: Error finding container 2d300712e711b03830289edf5cb38a7ecef41dad9f2269956fd7e6b3d7ada3b2: Status 404 returned error can't find the container with id 2d300712e711b03830289edf5cb38a7ecef41dad9f2269956fd7e6b3d7ada3b2
Oct 30 20:38:14.563174 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.563122    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-pllmn"
Oct 30 20:38:14.572051 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:14.572003    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode584a74f_f82c_4a3f_81d6_0b93601980dd.slice/crio-09d3ec791aaa85a086644433a7fab033324b526a8aec145332363a8c00682d5f WatchSource:0}: Error finding container 09d3ec791aaa85a086644433a7fab033324b526a8aec145332363a8c00682d5f: Status 404 returned error can't find the container with id 09d3ec791aaa85a086644433a7fab033324b526a8aec145332363a8c00682d5f
Oct 30 20:38:14.575203 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:14.575173    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76fb0a52_8bca_4610_9891_605df420cc7a.slice/crio-7b9bba86d8bf288421f6b51c926a38211bfe44f6aca4598113aab535b8b73fb2 WatchSource:0}: Error finding container 7b9bba86d8bf288421f6b51c926a38211bfe44f6aca4598113aab535b8b73fb2: Status 404 returned error can't find the container with id 7b9bba86d8bf288421f6b51c926a38211bfe44f6aca4598113aab535b8b73fb2
Oct 30 20:38:14.578865 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:14.578837    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5907106_812a_440a_b47d_f1889569d3b0.slice/crio-3ed5d651d95aef72bfc44b6f7712e92302c3d32557963157ea4de9e36905665c WatchSource:0}: Error finding container 3ed5d651d95aef72bfc44b6f7712e92302c3d32557963157ea4de9e36905665c: Status 404 returned error can't find the container with id 3ed5d651d95aef72bfc44b6f7712e92302c3d32557963157ea4de9e36905665c
Oct 30 20:38:14.630554 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.630498    2583 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 20:38:14.810667 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.810589    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a03cdbf5-91a2-41eb-8c1e-3c0f40da2632-original-pull-secret\") pod \"global-pull-secret-syncer-xkfsj\" (UID: \"a03cdbf5-91a2-41eb-8c1e-3c0f40da2632\") " pod="kube-system/global-pull-secret-syncer-xkfsj"
Oct 30 20:38:14.810667 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.810635    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6fdde40e-3330-4b84-a470-6a7e6a536dce-metrics-certs\") pod \"network-metrics-daemon-5bww9\" (UID: \"6fdde40e-3330-4b84-a470-6a7e6a536dce\") " pod="openshift-multus/network-metrics-daemon-5bww9"
Oct 30 20:38:14.810865 ip-10-0-131-125 kubenswrapper[2583]: E1030 20:38:14.810760    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 20:38:14.810865 ip-10-0-131-125 kubenswrapper[2583]: E1030 20:38:14.810834    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6fdde40e-3330-4b84-a470-6a7e6a536dce-metrics-certs podName:6fdde40e-3330-4b84-a470-6a7e6a536dce nodeName:}" failed. No retries permitted until 2026-10-30 20:38:15.810812488 +0000 UTC m=+3.987495014 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6fdde40e-3330-4b84-a470-6a7e6a536dce-metrics-certs") pod "network-metrics-daemon-5bww9" (UID: "6fdde40e-3330-4b84-a470-6a7e6a536dce") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 20:38:14.810865 ip-10-0-131-125 kubenswrapper[2583]: E1030 20:38:14.810761    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 20:38:14.811002 ip-10-0-131-125 kubenswrapper[2583]: E1030 20:38:14.810920    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a03cdbf5-91a2-41eb-8c1e-3c0f40da2632-original-pull-secret podName:a03cdbf5-91a2-41eb-8c1e-3c0f40da2632 nodeName:}" failed. No retries permitted until 2026-10-30 20:38:15.810899693 +0000 UTC m=+3.987582238 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a03cdbf5-91a2-41eb-8c1e-3c0f40da2632-original-pull-secret") pod "global-pull-secret-syncer-xkfsj" (UID: "a03cdbf5-91a2-41eb-8c1e-3c0f40da2632") : object "kube-system"/"original-pull-secret" not registered
Oct 30 20:38:14.911223 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:14.911180    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqttz\" (UniqueName: \"kubernetes.io/projected/7957ff19-e0fc-42a7-967d-6e96e4a88094-kube-api-access-pqttz\") pod \"network-check-target-f4cch\" (UID: \"7957ff19-e0fc-42a7-967d-6e96e4a88094\") " pod="openshift-network-diagnostics/network-check-target-f4cch"
Oct 30 20:38:14.911428 ip-10-0-131-125 kubenswrapper[2583]: E1030 20:38:14.911386    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 20:38:14.911428 ip-10-0-131-125 kubenswrapper[2583]: E1030 20:38:14.911406    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 20:38:14.911428 ip-10-0-131-125 kubenswrapper[2583]: E1030 20:38:14.911419    2583 projected.go:196] Error preparing data for projected volume kube-api-access-pqttz for pod openshift-network-diagnostics/network-check-target-f4cch: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 20:38:14.911581 ip-10-0-131-125 kubenswrapper[2583]: E1030 20:38:14.911487    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7957ff19-e0fc-42a7-967d-6e96e4a88094-kube-api-access-pqttz podName:7957ff19-e0fc-42a7-967d-6e96e4a88094 nodeName:}" failed. No retries permitted until 2026-10-30 20:38:15.911466661 +0000 UTC m=+4.088149208 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-pqttz" (UniqueName: "kubernetes.io/projected/7957ff19-e0fc-42a7-967d-6e96e4a88094-kube-api-access-pqttz") pod "network-check-target-f4cch" (UID: "7957ff19-e0fc-42a7-967d-6e96e4a88094") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 20:38:15.255840 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:15.255239    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f4cch"
Oct 30 20:38:15.255840 ip-10-0-131-125 kubenswrapper[2583]: E1030 20:38:15.255417    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f4cch" podUID="7957ff19-e0fc-42a7-967d-6e96e4a88094"
Oct 30 20:38:15.257302 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:15.256948    2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 20:33:14 +0000 UTC" deadline="2028-04-21 12:04:07.974630934 +0000 UTC"
Oct 30 20:38:15.257302 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:15.257235    2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12927h25m52.71740236s"
Oct 30 20:38:15.280455 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:15.279634    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-125.us-east-2.compute.internal" event={"ID":"cbbd659d200641ef129d88b1e4575eca","Type":"ContainerStarted","Data":"ed50815f5f70867e399607ccb259261aba2d787f157a0366878321ab09b25a4b"}
Oct 30 20:38:15.287134 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:15.286950    2583 generic.go:356] "Generic (PLEG): container finished" podID="73fe153e2da84792f2cfb4a59468aa21" containerID="674c2550b650706629cff4ef93d995f87a11905a35a5a10439e42553230ab3b1" exitCode=0
Oct 30 20:38:15.287134 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:15.287035    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-125.us-east-2.compute.internal" event={"ID":"73fe153e2da84792f2cfb4a59468aa21","Type":"ContainerDied","Data":"674c2550b650706629cff4ef93d995f87a11905a35a5a10439e42553230ab3b1"}
Oct 30 20:38:15.296995 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:15.296964    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p5hfb" event={"ID":"e584a74f-f82c-4a3f-81d6-0b93601980dd","Type":"ContainerStarted","Data":"09d3ec791aaa85a086644433a7fab033324b526a8aec145332363a8c00682d5f"}
Oct 30 20:38:15.312280 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:15.312112    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mt2kl" event={"ID":"c3967ccc-990a-42f5-845e-cbeec5e18be0","Type":"ContainerStarted","Data":"56461b767faa0395c16b73aa678e90856e30180670d31d772bd981dd7a51ceb5"}
Oct 30 20:38:15.324641 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:15.323409    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-125.us-east-2.compute.internal" podStartSLOduration=2.323390904 podStartE2EDuration="2.323390904s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:38:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:38:15.303218206 +0000 UTC m=+3.479900760" watchObservedRunningTime="2026-10-30 20:38:15.323390904 +0000 UTC m=+3.500073457"
Oct 30 20:38:15.342895 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:15.342856    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-2kdx9" event={"ID":"880d0493-e39d-4d9c-b58b-aa48c33aeaef","Type":"ContainerStarted","Data":"228acd337e25401e4710ece12fd7e71c0d4581a209aef19e35790ee08ea50a0e"}
Oct 30 20:38:15.352518 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:15.352479    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-4kpcm" event={"ID":"c6467dfc-7d9c-4ba8-ac2e-2ce19a74a0d4","Type":"ContainerStarted","Data":"d8c56fe4c6222b749344a9775e4bb3134b1453511b9b5929dc9ea72f76dbe863"}
Oct 30 20:38:15.356022 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:15.355995    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-kxvqp" event={"ID":"ee639a0c-5f44-4091-a876-fe9109934f74","Type":"ContainerStarted","Data":"0bcc74b6507dc0209f85ad2a19b8912b241efe513eb5ad35491b1737ca8a262c"}
Oct 30 20:38:15.358863 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:15.358831    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-pllmn" event={"ID":"d5907106-812a-440a-b47d-f1889569d3b0","Type":"ContainerStarted","Data":"3ed5d651d95aef72bfc44b6f7712e92302c3d32557963157ea4de9e36905665c"}
Oct 30 20:38:15.367114 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:15.367079    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kxsjg" event={"ID":"76fb0a52-8bca-4610-9891-605df420cc7a","Type":"ContainerStarted","Data":"7b9bba86d8bf288421f6b51c926a38211bfe44f6aca4598113aab535b8b73fb2"}
Oct 30 20:38:15.381915 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:15.381870    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-9p5zk" event={"ID":"f831b1bf-4a1f-48b7-8242-85c063825277","Type":"ContainerStarted","Data":"2d300712e711b03830289edf5cb38a7ecef41dad9f2269956fd7e6b3d7ada3b2"}
Oct 30 20:38:15.398786 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:15.398743    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-qcrpl" event={"ID":"d585cbd8-cb89-4718-9af4-716f64cc2598","Type":"ContainerStarted","Data":"97605bd7a36f6cc05063a503084c058056dfa45b02a518afbf5402517fbc5e60"}
Oct 30 20:38:15.821338 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:15.821293    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a03cdbf5-91a2-41eb-8c1e-3c0f40da2632-original-pull-secret\") pod \"global-pull-secret-syncer-xkfsj\" (UID: \"a03cdbf5-91a2-41eb-8c1e-3c0f40da2632\") " pod="kube-system/global-pull-secret-syncer-xkfsj"
Oct 30 20:38:15.821564 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:15.821393    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6fdde40e-3330-4b84-a470-6a7e6a536dce-metrics-certs\") pod \"network-metrics-daemon-5bww9\" (UID: \"6fdde40e-3330-4b84-a470-6a7e6a536dce\") " pod="openshift-multus/network-metrics-daemon-5bww9"
Oct 30 20:38:15.821634 ip-10-0-131-125 kubenswrapper[2583]: E1030 20:38:15.821586    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 20:38:15.821691 ip-10-0-131-125 kubenswrapper[2583]: E1030 20:38:15.821653    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6fdde40e-3330-4b84-a470-6a7e6a536dce-metrics-certs podName:6fdde40e-3330-4b84-a470-6a7e6a536dce nodeName:}" failed. No retries permitted until 2026-10-30 20:38:17.82163355 +0000 UTC m=+5.998316091 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6fdde40e-3330-4b84-a470-6a7e6a536dce-metrics-certs") pod "network-metrics-daemon-5bww9" (UID: "6fdde40e-3330-4b84-a470-6a7e6a536dce") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 20:38:15.822135 ip-10-0-131-125 kubenswrapper[2583]: E1030 20:38:15.822114    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 20:38:15.822221 ip-10-0-131-125 kubenswrapper[2583]: E1030 20:38:15.822197    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a03cdbf5-91a2-41eb-8c1e-3c0f40da2632-original-pull-secret podName:a03cdbf5-91a2-41eb-8c1e-3c0f40da2632 nodeName:}" failed. No retries permitted until 2026-10-30 20:38:17.822160596 +0000 UTC m=+5.998843138 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a03cdbf5-91a2-41eb-8c1e-3c0f40da2632-original-pull-secret") pod "global-pull-secret-syncer-xkfsj" (UID: "a03cdbf5-91a2-41eb-8c1e-3c0f40da2632") : object "kube-system"/"original-pull-secret" not registered
Oct 30 20:38:15.922483 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:15.921804    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqttz\" (UniqueName: \"kubernetes.io/projected/7957ff19-e0fc-42a7-967d-6e96e4a88094-kube-api-access-pqttz\") pod \"network-check-target-f4cch\" (UID: \"7957ff19-e0fc-42a7-967d-6e96e4a88094\") " pod="openshift-network-diagnostics/network-check-target-f4cch"
Oct 30 20:38:15.922483 ip-10-0-131-125 kubenswrapper[2583]: E1030 20:38:15.921985    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 20:38:15.922483 ip-10-0-131-125 kubenswrapper[2583]: E1030 20:38:15.922005    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 20:38:15.922483 ip-10-0-131-125 kubenswrapper[2583]: E1030 20:38:15.922020    2583 projected.go:196] Error preparing data for projected volume kube-api-access-pqttz for pod openshift-network-diagnostics/network-check-target-f4cch: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 20:38:15.922483 ip-10-0-131-125 kubenswrapper[2583]: E1030 20:38:15.922087    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7957ff19-e0fc-42a7-967d-6e96e4a88094-kube-api-access-pqttz podName:7957ff19-e0fc-42a7-967d-6e96e4a88094 nodeName:}" failed. No retries permitted until 2026-10-30 20:38:17.922067865 +0000 UTC m=+6.098750410 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-pqttz" (UniqueName: "kubernetes.io/projected/7957ff19-e0fc-42a7-967d-6e96e4a88094-kube-api-access-pqttz") pod "network-check-target-f4cch" (UID: "7957ff19-e0fc-42a7-967d-6e96e4a88094") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 20:38:16.257504 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:16.255372    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5bww9"
Oct 30 20:38:16.257504 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:16.255469    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xkfsj"
Oct 30 20:38:16.257504 ip-10-0-131-125 kubenswrapper[2583]: E1030 20:38:16.255602    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xkfsj" podUID="a03cdbf5-91a2-41eb-8c1e-3c0f40da2632"
Oct 30 20:38:16.258151 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:16.257839    2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 20:33:14 +0000 UTC" deadline="2028-07-11 15:07:15.347319476 +0000 UTC"
Oct 30 20:38:16.258151 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:16.257869    2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14874h28m59.089454751s"
Oct 30 20:38:16.258151 ip-10-0-131-125 kubenswrapper[2583]: E1030 20:38:16.258002    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5bww9" podUID="6fdde40e-3330-4b84-a470-6a7e6a536dce"
Oct 30 20:38:16.418438 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:16.418207    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-125.us-east-2.compute.internal" event={"ID":"73fe153e2da84792f2cfb4a59468aa21","Type":"ContainerStarted","Data":"ca65d62205a83e451c0fc33652d8bcf3c328603d7c9bd565aaad8247efdac4e7"}
Oct 30 20:38:16.437554 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:16.437449    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-125.us-east-2.compute.internal" podStartSLOduration=3.437430421 podStartE2EDuration="3.437430421s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:38:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:38:16.436662911 +0000 UTC m=+4.613345474" watchObservedRunningTime="2026-10-30 20:38:16.437430421 +0000 UTC m=+4.614112971"
Oct 30 20:38:17.254875 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:17.254840    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f4cch"
Oct 30 20:38:17.255050 ip-10-0-131-125 kubenswrapper[2583]: E1030 20:38:17.254970    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f4cch" podUID="7957ff19-e0fc-42a7-967d-6e96e4a88094"
Oct 30 20:38:17.855526 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:17.854671    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a03cdbf5-91a2-41eb-8c1e-3c0f40da2632-original-pull-secret\") pod \"global-pull-secret-syncer-xkfsj\" (UID: \"a03cdbf5-91a2-41eb-8c1e-3c0f40da2632\") " pod="kube-system/global-pull-secret-syncer-xkfsj"
Oct 30 20:38:17.855526 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:17.854734    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6fdde40e-3330-4b84-a470-6a7e6a536dce-metrics-certs\") pod \"network-metrics-daemon-5bww9\" (UID: \"6fdde40e-3330-4b84-a470-6a7e6a536dce\") " pod="openshift-multus/network-metrics-daemon-5bww9"
Oct 30 20:38:17.855526 ip-10-0-131-125 kubenswrapper[2583]: E1030 20:38:17.854872    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 20:38:17.855526 ip-10-0-131-125 kubenswrapper[2583]: E1030 20:38:17.854942    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6fdde40e-3330-4b84-a470-6a7e6a536dce-metrics-certs podName:6fdde40e-3330-4b84-a470-6a7e6a536dce nodeName:}" failed. No retries permitted until 2026-10-30 20:38:21.854919568 +0000 UTC m=+10.031602102 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6fdde40e-3330-4b84-a470-6a7e6a536dce-metrics-certs") pod "network-metrics-daemon-5bww9" (UID: "6fdde40e-3330-4b84-a470-6a7e6a536dce") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 20:38:17.855526 ip-10-0-131-125 kubenswrapper[2583]: E1030 20:38:17.855367    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 20:38:17.855526 ip-10-0-131-125 kubenswrapper[2583]: E1030 20:38:17.855438    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a03cdbf5-91a2-41eb-8c1e-3c0f40da2632-original-pull-secret podName:a03cdbf5-91a2-41eb-8c1e-3c0f40da2632 nodeName:}" failed. No retries permitted until 2026-10-30 20:38:21.855421811 +0000 UTC m=+10.032104343 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a03cdbf5-91a2-41eb-8c1e-3c0f40da2632-original-pull-secret") pod "global-pull-secret-syncer-xkfsj" (UID: "a03cdbf5-91a2-41eb-8c1e-3c0f40da2632") : object "kube-system"/"original-pull-secret" not registered
Oct 30 20:38:17.956653 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:17.955780    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqttz\" (UniqueName: \"kubernetes.io/projected/7957ff19-e0fc-42a7-967d-6e96e4a88094-kube-api-access-pqttz\") pod \"network-check-target-f4cch\" (UID: \"7957ff19-e0fc-42a7-967d-6e96e4a88094\") " pod="openshift-network-diagnostics/network-check-target-f4cch"
Oct 30 20:38:17.956653 ip-10-0-131-125 kubenswrapper[2583]: E1030 20:38:17.956032    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 20:38:17.956653 ip-10-0-131-125 kubenswrapper[2583]: E1030 20:38:17.956060    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 20:38:17.956653 ip-10-0-131-125 kubenswrapper[2583]: E1030 20:38:17.956076    2583 projected.go:196] Error preparing data for projected volume kube-api-access-pqttz for pod openshift-network-diagnostics/network-check-target-f4cch: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 20:38:17.956653 ip-10-0-131-125 kubenswrapper[2583]: E1030 20:38:17.956146    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7957ff19-e0fc-42a7-967d-6e96e4a88094-kube-api-access-pqttz podName:7957ff19-e0fc-42a7-967d-6e96e4a88094 nodeName:}" failed. No retries permitted until 2026-10-30 20:38:21.956125409 +0000 UTC m=+10.132807955 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-pqttz" (UniqueName: "kubernetes.io/projected/7957ff19-e0fc-42a7-967d-6e96e4a88094-kube-api-access-pqttz") pod "network-check-target-f4cch" (UID: "7957ff19-e0fc-42a7-967d-6e96e4a88094") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 20:38:18.256344 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:18.255524    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5bww9"
Oct 30 20:38:18.256344 ip-10-0-131-125 kubenswrapper[2583]: E1030 20:38:18.255678    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5bww9" podUID="6fdde40e-3330-4b84-a470-6a7e6a536dce"
Oct 30 20:38:18.256344 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:18.256142    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xkfsj"
Oct 30 20:38:18.256344 ip-10-0-131-125 kubenswrapper[2583]: E1030 20:38:18.256245    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xkfsj" podUID="a03cdbf5-91a2-41eb-8c1e-3c0f40da2632"
Oct 30 20:38:18.428045 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:18.428003    2583 generic.go:356] "Generic (PLEG): container finished" podID="e584a74f-f82c-4a3f-81d6-0b93601980dd" containerID="ae29407fdc5eb0abdc0ebf1bb4f40785283580fd62971ffa93deef75ba422d49" exitCode=0
Oct 30 20:38:18.428205 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:18.428061    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p5hfb" event={"ID":"e584a74f-f82c-4a3f-81d6-0b93601980dd","Type":"ContainerDied","Data":"ae29407fdc5eb0abdc0ebf1bb4f40785283580fd62971ffa93deef75ba422d49"}
Oct 30 20:38:19.255797 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:19.255760    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f4cch"
Oct 30 20:38:19.256277 ip-10-0-131-125 kubenswrapper[2583]: E1030 20:38:19.256005    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f4cch" podUID="7957ff19-e0fc-42a7-967d-6e96e4a88094"
Oct 30 20:38:19.435365 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:19.435314    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-qcrpl" event={"ID":"d585cbd8-cb89-4718-9af4-716f64cc2598","Type":"ContainerStarted","Data":"3276d84bd04d0475f7c6a84497f29d8d9ebfeca20b3cffac6b0a4ddbec115024"}
Oct 30 20:38:20.258139 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:20.257472    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xkfsj"
Oct 30 20:38:20.258139 ip-10-0-131-125 kubenswrapper[2583]: E1030 20:38:20.257615    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xkfsj" podUID="a03cdbf5-91a2-41eb-8c1e-3c0f40da2632"
Oct 30 20:38:20.258139 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:20.257486    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5bww9"
Oct 30 20:38:20.258139 ip-10-0-131-125 kubenswrapper[2583]: E1030 20:38:20.258060    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5bww9" podUID="6fdde40e-3330-4b84-a470-6a7e6a536dce"
Oct 30 20:38:20.440197 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:20.440160    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-2kdx9" event={"ID":"880d0493-e39d-4d9c-b58b-aa48c33aeaef","Type":"ContainerStarted","Data":"d7b3d3a014f68106661e89553655a2a43370708f50feff11bf0ffe866c2cff3c"}
Oct 30 20:38:20.442846 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:20.442800    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kxsjg" event={"ID":"76fb0a52-8bca-4610-9891-605df420cc7a","Type":"ContainerStarted","Data":"b8704d395658c5816b18db199ba8dc3377aa828679540d5de6886c8f8d87d65d"}
Oct 30 20:38:20.457212 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:20.457162    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-qcrpl" podStartSLOduration=4.249358686 podStartE2EDuration="8.457148013s" totalImagesPullingTime="4.207789327s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:38:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:38:19.45471669 +0000 UTC m=+7.631399239" watchObservedRunningTime="2026-10-30 20:38:20.457148013 +0000 UTC m=+8.633830560"
Oct 30 20:38:20.457367 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:20.457294    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-2kdx9" podStartSLOduration=3.113190116 podStartE2EDuration="8.457286668s" totalImagesPullingTime="5.344096552s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:38:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:38:20.457272387 +0000 UTC m=+8.633954935" watchObservedRunningTime="2026-10-30 20:38:20.457286668 +0000 UTC m=+8.633969219"
Oct 30 20:38:21.007147 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:21.007100    2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-qcrpl"
Oct 30 20:38:21.101225 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:21.101192    2583 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-qcrpl"
Oct 30 20:38:21.102003 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:21.101755    2583 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-qcrpl"
Oct 30 20:38:21.255808 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:21.255724    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f4cch"
Oct 30 20:38:21.255971 ip-10-0-131-125 kubenswrapper[2583]: E1030 20:38:21.255859    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f4cch" podUID="7957ff19-e0fc-42a7-967d-6e96e4a88094"
Oct 30 20:38:21.446417 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:21.446203    2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-qcrpl"
Oct 30 20:38:21.892006 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:21.891971    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a03cdbf5-91a2-41eb-8c1e-3c0f40da2632-original-pull-secret\") pod \"global-pull-secret-syncer-xkfsj\" (UID: \"a03cdbf5-91a2-41eb-8c1e-3c0f40da2632\") " pod="kube-system/global-pull-secret-syncer-xkfsj"
Oct 30 20:38:21.892171 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:21.892021    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6fdde40e-3330-4b84-a470-6a7e6a536dce-metrics-certs\") pod \"network-metrics-daemon-5bww9\" (UID: \"6fdde40e-3330-4b84-a470-6a7e6a536dce\") " pod="openshift-multus/network-metrics-daemon-5bww9"
Oct 30 20:38:21.892171 ip-10-0-131-125 kubenswrapper[2583]: E1030 20:38:21.892109    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 20:38:21.892171 ip-10-0-131-125 kubenswrapper[2583]: E1030 20:38:21.892140    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 20:38:21.892363 ip-10-0-131-125 kubenswrapper[2583]: E1030 20:38:21.892187    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a03cdbf5-91a2-41eb-8c1e-3c0f40da2632-original-pull-secret podName:a03cdbf5-91a2-41eb-8c1e-3c0f40da2632 nodeName:}" failed. No retries permitted until 2026-10-30 20:38:29.892166498 +0000 UTC m=+18.068849028 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a03cdbf5-91a2-41eb-8c1e-3c0f40da2632-original-pull-secret") pod "global-pull-secret-syncer-xkfsj" (UID: "a03cdbf5-91a2-41eb-8c1e-3c0f40da2632") : object "kube-system"/"original-pull-secret" not registered
Oct 30 20:38:21.892363 ip-10-0-131-125 kubenswrapper[2583]: E1030 20:38:21.892207    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6fdde40e-3330-4b84-a470-6a7e6a536dce-metrics-certs podName:6fdde40e-3330-4b84-a470-6a7e6a536dce nodeName:}" failed. No retries permitted until 2026-10-30 20:38:29.89219761 +0000 UTC m=+18.068880141 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6fdde40e-3330-4b84-a470-6a7e6a536dce-metrics-certs") pod "network-metrics-daemon-5bww9" (UID: "6fdde40e-3330-4b84-a470-6a7e6a536dce") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 20:38:21.992394 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:21.992310    2583 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 20:38:21.992608 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:21.992590    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqttz\" (UniqueName: \"kubernetes.io/projected/7957ff19-e0fc-42a7-967d-6e96e4a88094-kube-api-access-pqttz\") pod \"network-check-target-f4cch\" (UID: \"7957ff19-e0fc-42a7-967d-6e96e4a88094\") " pod="openshift-network-diagnostics/network-check-target-f4cch"
Oct 30 20:38:21.992713 ip-10-0-131-125 kubenswrapper[2583]: E1030 20:38:21.992698    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 20:38:21.992768 ip-10-0-131-125 kubenswrapper[2583]: E1030 20:38:21.992718    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 20:38:21.992768 ip-10-0-131-125 kubenswrapper[2583]: E1030 20:38:21.992727    2583 projected.go:196] Error preparing data for projected volume kube-api-access-pqttz for pod openshift-network-diagnostics/network-check-target-f4cch: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 20:38:21.992858 ip-10-0-131-125 kubenswrapper[2583]: E1030 20:38:21.992775    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7957ff19-e0fc-42a7-967d-6e96e4a88094-kube-api-access-pqttz podName:7957ff19-e0fc-42a7-967d-6e96e4a88094 nodeName:}" failed. No retries permitted until 2026-10-30 20:38:29.992760573 +0000 UTC m=+18.169443111 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-pqttz" (UniqueName: "kubernetes.io/projected/7957ff19-e0fc-42a7-967d-6e96e4a88094-kube-api-access-pqttz") pod "network-check-target-f4cch" (UID: "7957ff19-e0fc-42a7-967d-6e96e4a88094") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 20:38:22.260195 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:22.257630    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5bww9"
Oct 30 20:38:22.260195 ip-10-0-131-125 kubenswrapper[2583]: E1030 20:38:22.257852    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5bww9" podUID="6fdde40e-3330-4b84-a470-6a7e6a536dce"
Oct 30 20:38:22.260195 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:22.258228    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xkfsj"
Oct 30 20:38:22.260195 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:22.259261    2583 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T20:38:21.992350405Z","UUID":"a51bdde8-f141-4c2c-862e-d20a4916ba1c","Handler":null,"Name":"","Endpoint":""}
Oct 30 20:38:22.260814 ip-10-0-131-125 kubenswrapper[2583]: E1030 20:38:22.258318    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xkfsj" podUID="a03cdbf5-91a2-41eb-8c1e-3c0f40da2632"
Oct 30 20:38:22.261670 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:22.261125    2583 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 20:38:22.261670 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:22.261152    2583 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 20:38:22.453014 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:22.452957    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-kxvqp" event={"ID":"ee639a0c-5f44-4091-a876-fe9109934f74","Type":"ContainerStarted","Data":"e7e1362c15d502f7f4c182d5aa58c18600d19d1e0e80f6a6426ea1d0e1d77820"}
Oct 30 20:38:22.461450 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:22.461407    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kxsjg" event={"ID":"76fb0a52-8bca-4610-9891-605df420cc7a","Type":"ContainerStarted","Data":"691d91a80c7d0286456c0370e6047ccf8170b6c793c76c10335d159a6967c1c4"}
Oct 30 20:38:23.255404 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:23.255371    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f4cch"
Oct 30 20:38:23.255549 ip-10-0-131-125 kubenswrapper[2583]: E1030 20:38:23.255501    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f4cch" podUID="7957ff19-e0fc-42a7-967d-6e96e4a88094"
Oct 30 20:38:23.466011 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:23.465770    2583 generic.go:356] "Generic (PLEG): container finished" podID="e584a74f-f82c-4a3f-81d6-0b93601980dd" containerID="a5894271d75c9731ad7a8d91d9e8fb1fb8a131095dbdc896e06047c8b0ba638c" exitCode=0
Oct 30 20:38:23.466011 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:23.465953    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p5hfb" event={"ID":"e584a74f-f82c-4a3f-81d6-0b93601980dd","Type":"ContainerDied","Data":"a5894271d75c9731ad7a8d91d9e8fb1fb8a131095dbdc896e06047c8b0ba638c"}
Oct 30 20:38:23.467654 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:23.467624    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-4kpcm" event={"ID":"c6467dfc-7d9c-4ba8-ac2e-2ce19a74a0d4","Type":"ContainerStarted","Data":"e28ac09d8d5ad5c02c5f879914f62bae9d2a8bde1e07812a3eeb9d32ecf3004f"}
Oct 30 20:38:23.470738 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:23.470711    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kxsjg" event={"ID":"76fb0a52-8bca-4610-9891-605df420cc7a","Type":"ContainerStarted","Data":"17c03e021bf0f96d23efd891c4ea33b92e84fa4ce64361658a6d949e182eeed4"}
Oct 30 20:38:23.491637 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:23.491583    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-kxvqp" podStartSLOduration=3.695291561 podStartE2EDuration="11.491571117s" totalImagesPullingTime="7.796279556s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:38:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:38:22.475682603 +0000 UTC m=+10.652365151" watchObservedRunningTime="2026-10-30 20:38:23.491571117 +0000 UTC m=+11.668253664"
Oct 30 20:38:23.517385 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:23.517309    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kxsjg" podStartSLOduration=3.265809713 podStartE2EDuration="11.517290008s" totalImagesPullingTime="8.251480295s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:38:12 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:38:23.516730227 +0000 UTC m=+11.693412798" watchObservedRunningTime="2026-10-30 20:38:23.517290008 +0000 UTC m=+11.693972556"
Oct 30 20:38:23.553615 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:23.553551    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-4kpcm" podStartSLOduration=3.399077253 podStartE2EDuration="11.553537469s" totalImagesPullingTime="8.154460216s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:38:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:38:23.553433553 +0000 UTC m=+11.730116102" watchObservedRunningTime="2026-10-30 20:38:23.553537469 +0000 UTC m=+11.730220017"
Oct 30 20:38:24.255497 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:24.255458    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5bww9"
Oct 30 20:38:24.255657 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:24.255458    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xkfsj"
Oct 30 20:38:24.255657 ip-10-0-131-125 kubenswrapper[2583]: E1030 20:38:24.255601    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5bww9" podUID="6fdde40e-3330-4b84-a470-6a7e6a536dce"
Oct 30 20:38:24.255657 ip-10-0-131-125 kubenswrapper[2583]: E1030 20:38:24.255624    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xkfsj" podUID="a03cdbf5-91a2-41eb-8c1e-3c0f40da2632"
Oct 30 20:38:24.475604 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:24.475526    2583 generic.go:356] "Generic (PLEG): container finished" podID="e584a74f-f82c-4a3f-81d6-0b93601980dd" containerID="dd6a5a9517561387d40724145a33bba7fc7bbe9d81a0fa650dc9bfea18a1947d" exitCode=0
Oct 30 20:38:24.475986 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:24.475642    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p5hfb" event={"ID":"e584a74f-f82c-4a3f-81d6-0b93601980dd","Type":"ContainerDied","Data":"dd6a5a9517561387d40724145a33bba7fc7bbe9d81a0fa650dc9bfea18a1947d"}
Oct 30 20:38:25.255212 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:25.255179    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f4cch"
Oct 30 20:38:25.255368 ip-10-0-131-125 kubenswrapper[2583]: E1030 20:38:25.255284    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f4cch" podUID="7957ff19-e0fc-42a7-967d-6e96e4a88094"
Oct 30 20:38:25.479510 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:25.479480    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-pllmn" event={"ID":"d5907106-812a-440a-b47d-f1889569d3b0","Type":"ContainerStarted","Data":"32d2d958c1ca41a75730c9f339f503114c9bf6a5254beb648aac9d4ae72a3594"}
Oct 30 20:38:25.481876 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:25.481847    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-9p5zk" event={"ID":"f831b1bf-4a1f-48b7-8242-85c063825277","Type":"ContainerStarted","Data":"6e761dcdb0f9cdace89233577699c6925cf968fe21e89dc4f9805effc217f711"}
Oct 30 20:38:25.500237 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:25.500177    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-pllmn" podStartSLOduration=2.99490754 podStartE2EDuration="13.500162744s" totalImagesPullingTime="10.505255204s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:38:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:38:25.500022194 +0000 UTC m=+13.676704742" watchObservedRunningTime="2026-10-30 20:38:25.500162744 +0000 UTC m=+13.676845293"
Oct 30 20:38:25.518465 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:25.518416    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-9p5zk" podStartSLOduration=5.189791319 podStartE2EDuration="13.518401863s" totalImagesPullingTime="8.328610544s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:38:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:38:25.51786778 +0000 UTC m=+13.694550329" watchObservedRunningTime="2026-10-30 20:38:25.518401863 +0000 UTC m=+13.695084409"
Oct 30 20:38:26.255191 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:26.255155    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xkfsj"
Oct 30 20:38:26.255398 ip-10-0-131-125 kubenswrapper[2583]: E1030 20:38:26.255276    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xkfsj" podUID="a03cdbf5-91a2-41eb-8c1e-3c0f40da2632"
Oct 30 20:38:26.255398 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:26.255305    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5bww9"
Oct 30 20:38:26.255398 ip-10-0-131-125 kubenswrapper[2583]: E1030 20:38:26.255378    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5bww9" podUID="6fdde40e-3330-4b84-a470-6a7e6a536dce"
Oct 30 20:38:26.487286 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:26.487247    2583 generic.go:356] "Generic (PLEG): container finished" podID="e584a74f-f82c-4a3f-81d6-0b93601980dd" containerID="276df7fca27c378acae9e76a6dae55982406f1c0ec0861c419827bbba91ce1af" exitCode=0
Oct 30 20:38:26.487972 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:26.487339    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p5hfb" event={"ID":"e584a74f-f82c-4a3f-81d6-0b93601980dd","Type":"ContainerDied","Data":"276df7fca27c378acae9e76a6dae55982406f1c0ec0861c419827bbba91ce1af"}
Oct 30 20:38:27.255254 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:27.255225    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f4cch"
Oct 30 20:38:27.255455 ip-10-0-131-125 kubenswrapper[2583]: E1030 20:38:27.255359    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f4cch" podUID="7957ff19-e0fc-42a7-967d-6e96e4a88094"
Oct 30 20:38:28.254773 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:28.254737    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xkfsj"
Oct 30 20:38:28.255451 ip-10-0-131-125 kubenswrapper[2583]: E1030 20:38:28.254881    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xkfsj" podUID="a03cdbf5-91a2-41eb-8c1e-3c0f40da2632"
Oct 30 20:38:28.255451 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:28.254944    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5bww9"
Oct 30 20:38:28.255451 ip-10-0-131-125 kubenswrapper[2583]: E1030 20:38:28.255088    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5bww9" podUID="6fdde40e-3330-4b84-a470-6a7e6a536dce"
Oct 30 20:38:28.495200 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:28.494978    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p5hfb" event={"ID":"e584a74f-f82c-4a3f-81d6-0b93601980dd","Type":"ContainerStarted","Data":"3aba57da98cc68bea670d9e7414c9400688d3c280c891ec42d6d23c535c9cccb"}
Oct 30 20:38:29.255360 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:29.255292    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f4cch"
Oct 30 20:38:29.255878 ip-10-0-131-125 kubenswrapper[2583]: E1030 20:38:29.255427    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f4cch" podUID="7957ff19-e0fc-42a7-967d-6e96e4a88094"
Oct 30 20:38:29.499238 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:29.499073    2583 generic.go:356] "Generic (PLEG): container finished" podID="e584a74f-f82c-4a3f-81d6-0b93601980dd" containerID="3aba57da98cc68bea670d9e7414c9400688d3c280c891ec42d6d23c535c9cccb" exitCode=0
Oct 30 20:38:29.499353 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:29.499153    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p5hfb" event={"ID":"e584a74f-f82c-4a3f-81d6-0b93601980dd","Type":"ContainerDied","Data":"3aba57da98cc68bea670d9e7414c9400688d3c280c891ec42d6d23c535c9cccb"}
Oct 30 20:38:29.501603 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:29.501582    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mt2kl" event={"ID":"c3967ccc-990a-42f5-845e-cbeec5e18be0","Type":"ContainerStarted","Data":"7fec62892783ab567d4f03400156a47b7cd2e0f7bb7931f1e0e5cc6d184ef642"}
Oct 30 20:38:29.501673 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:29.501637    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mt2kl" event={"ID":"c3967ccc-990a-42f5-845e-cbeec5e18be0","Type":"ContainerStarted","Data":"b0c4fe95dcb41e3778c17758e44007f8259f23da4c1a56c3d0559ae4c7c81945"}
Oct 30 20:38:29.501673 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:29.501665    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mt2kl" event={"ID":"c3967ccc-990a-42f5-845e-cbeec5e18be0","Type":"ContainerStarted","Data":"02abb44d8d84bf7790b13911e79905e197e27c2d98d20443dae62d7cace840e5"}
Oct 30 20:38:29.501746 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:29.501705    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mt2kl" event={"ID":"c3967ccc-990a-42f5-845e-cbeec5e18be0","Type":"ContainerStarted","Data":"0bab5a3c877a0ac94c05b15562bbb8f54fb6ea798f5f0733eac13a7ad595e615"}
Oct 30 20:38:29.955196 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:29.955092    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a03cdbf5-91a2-41eb-8c1e-3c0f40da2632-original-pull-secret\") pod \"global-pull-secret-syncer-xkfsj\" (UID: \"a03cdbf5-91a2-41eb-8c1e-3c0f40da2632\") " pod="kube-system/global-pull-secret-syncer-xkfsj"
Oct 30 20:38:29.955196 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:29.955148    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6fdde40e-3330-4b84-a470-6a7e6a536dce-metrics-certs\") pod \"network-metrics-daemon-5bww9\" (UID: \"6fdde40e-3330-4b84-a470-6a7e6a536dce\") " pod="openshift-multus/network-metrics-daemon-5bww9"
Oct 30 20:38:29.955470 ip-10-0-131-125 kubenswrapper[2583]: E1030 20:38:29.955256    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 20:38:29.955470 ip-10-0-131-125 kubenswrapper[2583]: E1030 20:38:29.955256    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 20:38:29.955470 ip-10-0-131-125 kubenswrapper[2583]: E1030 20:38:29.955319    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6fdde40e-3330-4b84-a470-6a7e6a536dce-metrics-certs podName:6fdde40e-3330-4b84-a470-6a7e6a536dce nodeName:}" failed. No retries permitted until 2026-10-30 20:38:45.955302526 +0000 UTC m=+34.131985066 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6fdde40e-3330-4b84-a470-6a7e6a536dce-metrics-certs") pod "network-metrics-daemon-5bww9" (UID: "6fdde40e-3330-4b84-a470-6a7e6a536dce") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 20:38:29.955470 ip-10-0-131-125 kubenswrapper[2583]: E1030 20:38:29.955408    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a03cdbf5-91a2-41eb-8c1e-3c0f40da2632-original-pull-secret podName:a03cdbf5-91a2-41eb-8c1e-3c0f40da2632 nodeName:}" failed. No retries permitted until 2026-10-30 20:38:45.95539607 +0000 UTC m=+34.132078595 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a03cdbf5-91a2-41eb-8c1e-3c0f40da2632-original-pull-secret") pod "global-pull-secret-syncer-xkfsj" (UID: "a03cdbf5-91a2-41eb-8c1e-3c0f40da2632") : object "kube-system"/"original-pull-secret" not registered
Oct 30 20:38:30.056137 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:30.056099    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqttz\" (UniqueName: \"kubernetes.io/projected/7957ff19-e0fc-42a7-967d-6e96e4a88094-kube-api-access-pqttz\") pod \"network-check-target-f4cch\" (UID: \"7957ff19-e0fc-42a7-967d-6e96e4a88094\") " pod="openshift-network-diagnostics/network-check-target-f4cch"
Oct 30 20:38:30.056316 ip-10-0-131-125 kubenswrapper[2583]: E1030 20:38:30.056229    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 20:38:30.056316 ip-10-0-131-125 kubenswrapper[2583]: E1030 20:38:30.056252    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 20:38:30.056316 ip-10-0-131-125 kubenswrapper[2583]: E1030 20:38:30.056261    2583 projected.go:196] Error preparing data for projected volume kube-api-access-pqttz for pod openshift-network-diagnostics/network-check-target-f4cch: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 20:38:30.056316 ip-10-0-131-125 kubenswrapper[2583]: E1030 20:38:30.056304    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7957ff19-e0fc-42a7-967d-6e96e4a88094-kube-api-access-pqttz podName:7957ff19-e0fc-42a7-967d-6e96e4a88094 nodeName:}" failed. No retries permitted until 2026-10-30 20:38:46.056292305 +0000 UTC m=+34.232974829 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-pqttz" (UniqueName: "kubernetes.io/projected/7957ff19-e0fc-42a7-967d-6e96e4a88094-kube-api-access-pqttz") pod "network-check-target-f4cch" (UID: "7957ff19-e0fc-42a7-967d-6e96e4a88094") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 20:38:30.258204 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:30.258122    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5bww9"
Oct 30 20:38:30.258204 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:30.258155    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xkfsj"
Oct 30 20:38:30.258659 ip-10-0-131-125 kubenswrapper[2583]: E1030 20:38:30.258245    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5bww9" podUID="6fdde40e-3330-4b84-a470-6a7e6a536dce"
Oct 30 20:38:30.258659 ip-10-0-131-125 kubenswrapper[2583]: E1030 20:38:30.258411    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xkfsj" podUID="a03cdbf5-91a2-41eb-8c1e-3c0f40da2632"
Oct 30 20:38:30.506242 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:30.506208    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mt2kl" event={"ID":"c3967ccc-990a-42f5-845e-cbeec5e18be0","Type":"ContainerStarted","Data":"94dede71b6502e00bfa09629d8ba087fe79af3b20aa27ed2b4909e97af8c2c1d"}
Oct 30 20:38:30.506412 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:30.506334    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mt2kl" event={"ID":"c3967ccc-990a-42f5-845e-cbeec5e18be0","Type":"ContainerStarted","Data":"1b31d0c6150535320c5f7fa46af72836d52ce0b7ddcbeb4deceed5cec14c6fb7"}
Oct 30 20:38:30.508873 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:30.508803    2583 generic.go:356] "Generic (PLEG): container finished" podID="e584a74f-f82c-4a3f-81d6-0b93601980dd" containerID="61b1535f8b0316807697c01cdbb41fcedd10ad8d2a7ac47aa13770305df7ed07" exitCode=0
Oct 30 20:38:30.508873 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:30.508834    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p5hfb" event={"ID":"e584a74f-f82c-4a3f-81d6-0b93601980dd","Type":"ContainerDied","Data":"61b1535f8b0316807697c01cdbb41fcedd10ad8d2a7ac47aa13770305df7ed07"}
Oct 30 20:38:31.255047 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:31.254996    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f4cch"
Oct 30 20:38:31.255202 ip-10-0-131-125 kubenswrapper[2583]: E1030 20:38:31.255116    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f4cch" podUID="7957ff19-e0fc-42a7-967d-6e96e4a88094"
Oct 30 20:38:31.513864 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:31.513785    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p5hfb" event={"ID":"e584a74f-f82c-4a3f-81d6-0b93601980dd","Type":"ContainerStarted","Data":"9aff87bea82baf46745a4a86411beba796fce40efc8e08da783d7664cc300147"}
Oct 30 20:38:31.541414 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:31.541369    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-p5hfb" podStartSLOduration=8.395915777999999 podStartE2EDuration="19.541353731s" totalImagesPullingTime="11.145437953s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:38:12 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:38:31.54085762 +0000 UTC m=+19.717540170" watchObservedRunningTime="2026-10-30 20:38:31.541353731 +0000 UTC m=+19.718036325"
Oct 30 20:38:32.257644 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:32.257614    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xkfsj"
Oct 30 20:38:32.257833 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:32.257622    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5bww9"
Oct 30 20:38:32.257833 ip-10-0-131-125 kubenswrapper[2583]: E1030 20:38:32.257720    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xkfsj" podUID="a03cdbf5-91a2-41eb-8c1e-3c0f40da2632"
Oct 30 20:38:32.257833 ip-10-0-131-125 kubenswrapper[2583]: E1030 20:38:32.257805    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5bww9" podUID="6fdde40e-3330-4b84-a470-6a7e6a536dce"
Oct 30 20:38:32.521969 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:32.521881    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mt2kl" event={"ID":"c3967ccc-990a-42f5-845e-cbeec5e18be0","Type":"ContainerStarted","Data":"f7fba52db6cf65a08b735bc025335ee76e37bf6ee2e2b9ecc7944d0866fa684d"}
Oct 30 20:38:33.254994 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:33.254956    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f4cch"
Oct 30 20:38:33.255195 ip-10-0-131-125 kubenswrapper[2583]: E1030 20:38:33.255077    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f4cch" podUID="7957ff19-e0fc-42a7-967d-6e96e4a88094"
Oct 30 20:38:34.255569 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:34.255532    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xkfsj"
Oct 30 20:38:34.255569 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:34.255555    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5bww9"
Oct 30 20:38:34.256310 ip-10-0-131-125 kubenswrapper[2583]: E1030 20:38:34.255640    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xkfsj" podUID="a03cdbf5-91a2-41eb-8c1e-3c0f40da2632"
Oct 30 20:38:34.256310 ip-10-0-131-125 kubenswrapper[2583]: E1030 20:38:34.255763    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5bww9" podUID="6fdde40e-3330-4b84-a470-6a7e6a536dce"
Oct 30 20:38:34.530229 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:34.530150    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mt2kl" event={"ID":"c3967ccc-990a-42f5-845e-cbeec5e18be0","Type":"ContainerStarted","Data":"2e011919ccdb1ef1650e3a1712e04a1c98daf05990b192258e3b6b0824c7098a"}
Oct 30 20:38:34.530426 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:34.530406    2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mt2kl"
Oct 30 20:38:34.546847 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:34.546669    2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mt2kl"
Oct 30 20:38:34.562965 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:34.562917    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-mt2kl" podStartSLOduration=7.848821581 podStartE2EDuration="22.562894637s" totalImagesPullingTime="14.714073056s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:38:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:38:34.562828797 +0000 UTC m=+22.739511343" watchObservedRunningTime="2026-10-30 20:38:34.562894637 +0000 UTC m=+22.739577178"
Oct 30 20:38:35.255409 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:35.255382    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f4cch"
Oct 30 20:38:35.255587 ip-10-0-131-125 kubenswrapper[2583]: E1030 20:38:35.255494    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f4cch" podUID="7957ff19-e0fc-42a7-967d-6e96e4a88094"
Oct 30 20:38:35.532869 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:35.532762    2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mt2kl"
Oct 30 20:38:35.532869 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:35.532856    2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mt2kl"
Oct 30 20:38:35.549874 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:35.549850    2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mt2kl"
Oct 30 20:38:35.591354 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:35.591292    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-xkfsj"]
Oct 30 20:38:35.591533 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:35.591479    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xkfsj"
Oct 30 20:38:35.591637 ip-10-0-131-125 kubenswrapper[2583]: E1030 20:38:35.591609    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xkfsj" podUID="a03cdbf5-91a2-41eb-8c1e-3c0f40da2632"
Oct 30 20:38:35.592004 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:35.591983    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-f4cch"]
Oct 30 20:38:35.592410 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:35.592077    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f4cch"
Oct 30 20:38:35.592410 ip-10-0-131-125 kubenswrapper[2583]: E1030 20:38:35.592146    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f4cch" podUID="7957ff19-e0fc-42a7-967d-6e96e4a88094"
Oct 30 20:38:35.592688 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:35.592662    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-5bww9"]
Oct 30 20:38:35.592776 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:35.592759    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5bww9"
Oct 30 20:38:35.592848 ip-10-0-131-125 kubenswrapper[2583]: E1030 20:38:35.592832    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5bww9" podUID="6fdde40e-3330-4b84-a470-6a7e6a536dce"
Oct 30 20:38:37.255725 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:37.255691    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5bww9"
Oct 30 20:38:37.256267 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:37.255691    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f4cch"
Oct 30 20:38:37.256267 ip-10-0-131-125 kubenswrapper[2583]: E1030 20:38:37.255808    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5bww9" podUID="6fdde40e-3330-4b84-a470-6a7e6a536dce"
Oct 30 20:38:37.256267 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:37.255690    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xkfsj"
Oct 30 20:38:37.256267 ip-10-0-131-125 kubenswrapper[2583]: E1030 20:38:37.255867    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f4cch" podUID="7957ff19-e0fc-42a7-967d-6e96e4a88094"
Oct 30 20:38:37.256267 ip-10-0-131-125 kubenswrapper[2583]: E1030 20:38:37.255961    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xkfsj" podUID="a03cdbf5-91a2-41eb-8c1e-3c0f40da2632"
Oct 30 20:38:39.255692 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:39.255655    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5bww9"
Oct 30 20:38:39.255692 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:39.255674    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f4cch"
Oct 30 20:38:39.256170 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:39.255756    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xkfsj"
Oct 30 20:38:39.256170 ip-10-0-131-125 kubenswrapper[2583]: E1030 20:38:39.255770    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5bww9" podUID="6fdde40e-3330-4b84-a470-6a7e6a536dce"
Oct 30 20:38:39.256170 ip-10-0-131-125 kubenswrapper[2583]: E1030 20:38:39.255828    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f4cch" podUID="7957ff19-e0fc-42a7-967d-6e96e4a88094"
Oct 30 20:38:39.256170 ip-10-0-131-125 kubenswrapper[2583]: E1030 20:38:39.255915    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xkfsj" podUID="a03cdbf5-91a2-41eb-8c1e-3c0f40da2632"
Oct 30 20:38:41.255678 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:41.255642    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xkfsj"
Oct 30 20:38:41.256154 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:41.255720    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5bww9"
Oct 30 20:38:41.256154 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:41.255747    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f4cch"
Oct 30 20:38:41.256154 ip-10-0-131-125 kubenswrapper[2583]: E1030 20:38:41.255858    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5bww9" podUID="6fdde40e-3330-4b84-a470-6a7e6a536dce"
Oct 30 20:38:41.256154 ip-10-0-131-125 kubenswrapper[2583]: E1030 20:38:41.255889    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f4cch" podUID="7957ff19-e0fc-42a7-967d-6e96e4a88094"
Oct 30 20:38:41.256154 ip-10-0-131-125 kubenswrapper[2583]: E1030 20:38:41.255967    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xkfsj" podUID="a03cdbf5-91a2-41eb-8c1e-3c0f40da2632"
Oct 30 20:38:42.059661 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:42.059448    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-125.us-east-2.compute.internal" event="NodeReady"
Oct 30 20:38:42.059791 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:42.059719    2583 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Oct 30 20:38:42.116646 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:42.116611    2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-q2wtm"]
Oct 30 20:38:42.120183 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:42.120155    2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-fblmm"]
Oct 30 20:38:42.120434 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:42.120276    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-q2wtm"
Oct 30 20:38:42.123700 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:42.123672    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-fblmm"
Oct 30 20:38:42.128847 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:42.128566    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 20:38:42.128847 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:42.128693    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 20:38:42.128847 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:42.128819    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 20:38:42.129045 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:42.128849    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 20:38:42.129045 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:42.128870    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-kxrnf\""
Oct 30 20:38:42.129045 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:42.129010    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-gk5pt\""
Oct 30 20:38:42.129184 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:42.129112    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 20:38:42.129644 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:42.129624    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-q2wtm"]
Oct 30 20:38:42.132502 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:42.132424    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-fblmm"]
Oct 30 20:38:42.243196 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:42.243155    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30561fcc-3184-4116-b179-18941ff26a20-tmp-dir\") pod \"dns-default-fblmm\" (UID: \"30561fcc-3184-4116-b179-18941ff26a20\") " pod="openshift-dns/dns-default-fblmm"
Oct 30 20:38:42.243196 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:42.243199    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-td58z\" (UniqueName: \"kubernetes.io/projected/30561fcc-3184-4116-b179-18941ff26a20-kube-api-access-td58z\") pod \"dns-default-fblmm\" (UID: \"30561fcc-3184-4116-b179-18941ff26a20\") " pod="openshift-dns/dns-default-fblmm"
Oct 30 20:38:42.243457 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:42.243345    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/30561fcc-3184-4116-b179-18941ff26a20-config-volume\") pod \"dns-default-fblmm\" (UID: \"30561fcc-3184-4116-b179-18941ff26a20\") " pod="openshift-dns/dns-default-fblmm"
Oct 30 20:38:42.243457 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:42.243383    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/30561fcc-3184-4116-b179-18941ff26a20-metrics-tls\") pod \"dns-default-fblmm\" (UID: \"30561fcc-3184-4116-b179-18941ff26a20\") " pod="openshift-dns/dns-default-fblmm"
Oct 30 20:38:42.243457 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:42.243401    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/747c15be-3782-4065-a6dc-73d9fe7a4b4f-cert\") pod \"ingress-canary-q2wtm\" (UID: \"747c15be-3782-4065-a6dc-73d9fe7a4b4f\") " pod="openshift-ingress-canary/ingress-canary-q2wtm"
Oct 30 20:38:42.243457 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:42.243439    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxcww\" (UniqueName: \"kubernetes.io/projected/747c15be-3782-4065-a6dc-73d9fe7a4b4f-kube-api-access-kxcww\") pod \"ingress-canary-q2wtm\" (UID: \"747c15be-3782-4065-a6dc-73d9fe7a4b4f\") " pod="openshift-ingress-canary/ingress-canary-q2wtm"
Oct 30 20:38:42.343897 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:42.343818    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxcww\" (UniqueName: \"kubernetes.io/projected/747c15be-3782-4065-a6dc-73d9fe7a4b4f-kube-api-access-kxcww\") pod \"ingress-canary-q2wtm\" (UID: \"747c15be-3782-4065-a6dc-73d9fe7a4b4f\") " pod="openshift-ingress-canary/ingress-canary-q2wtm"
Oct 30 20:38:42.343897 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:42.343880    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30561fcc-3184-4116-b179-18941ff26a20-tmp-dir\") pod \"dns-default-fblmm\" (UID: \"30561fcc-3184-4116-b179-18941ff26a20\") " pod="openshift-dns/dns-default-fblmm"
Oct 30 20:38:42.344653 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:42.344002    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-td58z\" (UniqueName: \"kubernetes.io/projected/30561fcc-3184-4116-b179-18941ff26a20-kube-api-access-td58z\") pod \"dns-default-fblmm\" (UID: \"30561fcc-3184-4116-b179-18941ff26a20\") " pod="openshift-dns/dns-default-fblmm"
Oct 30 20:38:42.344653 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:42.344061    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/30561fcc-3184-4116-b179-18941ff26a20-config-volume\") pod \"dns-default-fblmm\" (UID: \"30561fcc-3184-4116-b179-18941ff26a20\") " pod="openshift-dns/dns-default-fblmm"
Oct 30 20:38:42.344653 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:42.344086    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/30561fcc-3184-4116-b179-18941ff26a20-metrics-tls\") pod \"dns-default-fblmm\" (UID: \"30561fcc-3184-4116-b179-18941ff26a20\") " pod="openshift-dns/dns-default-fblmm"
Oct 30 20:38:42.344653 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:42.344114    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/747c15be-3782-4065-a6dc-73d9fe7a4b4f-cert\") pod \"ingress-canary-q2wtm\" (UID: \"747c15be-3782-4065-a6dc-73d9fe7a4b4f\") " pod="openshift-ingress-canary/ingress-canary-q2wtm"
Oct 30 20:38:42.344653 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:42.344191    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30561fcc-3184-4116-b179-18941ff26a20-tmp-dir\") pod \"dns-default-fblmm\" (UID: \"30561fcc-3184-4116-b179-18941ff26a20\") " pod="openshift-dns/dns-default-fblmm"
Oct 30 20:38:42.344653 ip-10-0-131-125 kubenswrapper[2583]: E1030 20:38:42.344238    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 20:38:42.344653 ip-10-0-131-125 kubenswrapper[2583]: E1030 20:38:42.344319    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30561fcc-3184-4116-b179-18941ff26a20-metrics-tls podName:30561fcc-3184-4116-b179-18941ff26a20 nodeName:}" failed. No retries permitted until 2026-10-30 20:38:42.844298813 +0000 UTC m=+31.020981354 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/30561fcc-3184-4116-b179-18941ff26a20-metrics-tls") pod "dns-default-fblmm" (UID: "30561fcc-3184-4116-b179-18941ff26a20") : secret "dns-default-metrics-tls" not found
Oct 30 20:38:42.344653 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:42.344581    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/30561fcc-3184-4116-b179-18941ff26a20-config-volume\") pod \"dns-default-fblmm\" (UID: \"30561fcc-3184-4116-b179-18941ff26a20\") " pod="openshift-dns/dns-default-fblmm"
Oct 30 20:38:42.348562 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:42.348538    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/747c15be-3782-4065-a6dc-73d9fe7a4b4f-cert\") pod \"ingress-canary-q2wtm\" (UID: \"747c15be-3782-4065-a6dc-73d9fe7a4b4f\") " pod="openshift-ingress-canary/ingress-canary-q2wtm"
Oct 30 20:38:42.358655 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:42.358629    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxcww\" (UniqueName: \"kubernetes.io/projected/747c15be-3782-4065-a6dc-73d9fe7a4b4f-kube-api-access-kxcww\") pod \"ingress-canary-q2wtm\" (UID: \"747c15be-3782-4065-a6dc-73d9fe7a4b4f\") " pod="openshift-ingress-canary/ingress-canary-q2wtm"
Oct 30 20:38:42.358750 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:42.358709    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-td58z\" (UniqueName: \"kubernetes.io/projected/30561fcc-3184-4116-b179-18941ff26a20-kube-api-access-td58z\") pod \"dns-default-fblmm\" (UID: \"30561fcc-3184-4116-b179-18941ff26a20\") " pod="openshift-dns/dns-default-fblmm"
Oct 30 20:38:42.438971 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:42.438933    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-q2wtm"
Oct 30 20:38:42.723921 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:42.723886    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-q2wtm"]
Oct 30 20:38:42.728488 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:42.728462    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod747c15be_3782_4065_a6dc_73d9fe7a4b4f.slice/crio-a594a0250290aea1b704ebd3a04e2f28875c682787b876d44d14c101b9e9f03b WatchSource:0}: Error finding container a594a0250290aea1b704ebd3a04e2f28875c682787b876d44d14c101b9e9f03b: Status 404 returned error can't find the container with id a594a0250290aea1b704ebd3a04e2f28875c682787b876d44d14c101b9e9f03b
Oct 30 20:38:42.848004 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:42.847965    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/30561fcc-3184-4116-b179-18941ff26a20-metrics-tls\") pod \"dns-default-fblmm\" (UID: \"30561fcc-3184-4116-b179-18941ff26a20\") " pod="openshift-dns/dns-default-fblmm"
Oct 30 20:38:42.848177 ip-10-0-131-125 kubenswrapper[2583]: E1030 20:38:42.848116    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 20:38:42.848220 ip-10-0-131-125 kubenswrapper[2583]: E1030 20:38:42.848179    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30561fcc-3184-4116-b179-18941ff26a20-metrics-tls podName:30561fcc-3184-4116-b179-18941ff26a20 nodeName:}" failed. No retries permitted until 2026-10-30 20:38:43.848163401 +0000 UTC m=+32.024845940 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/30561fcc-3184-4116-b179-18941ff26a20-metrics-tls") pod "dns-default-fblmm" (UID: "30561fcc-3184-4116-b179-18941ff26a20") : secret "dns-default-metrics-tls" not found
Oct 30 20:38:43.254740 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:43.254705    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xkfsj"
Oct 30 20:38:43.254740 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:43.254727    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f4cch"
Oct 30 20:38:43.254933 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:43.254718    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5bww9"
Oct 30 20:38:43.258905 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:43.258881    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-r8zbf\""
Oct 30 20:38:43.260084 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:43.259879    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6zzqh\""
Oct 30 20:38:43.260084 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:43.259966    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-q55wr\""
Oct 30 20:38:43.260084 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:43.260006    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 20:38:43.260084 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:43.260008    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 20:38:43.260084 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:43.260054    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 20:38:43.260084 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:43.260070    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 20:38:43.551610 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:43.551522    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-q2wtm" event={"ID":"747c15be-3782-4065-a6dc-73d9fe7a4b4f","Type":"ContainerStarted","Data":"a594a0250290aea1b704ebd3a04e2f28875c682787b876d44d14c101b9e9f03b"}
Oct 30 20:38:43.854120 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:43.854037    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/30561fcc-3184-4116-b179-18941ff26a20-metrics-tls\") pod \"dns-default-fblmm\" (UID: \"30561fcc-3184-4116-b179-18941ff26a20\") " pod="openshift-dns/dns-default-fblmm"
Oct 30 20:38:43.854272 ip-10-0-131-125 kubenswrapper[2583]: E1030 20:38:43.854199    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 20:38:43.854272 ip-10-0-131-125 kubenswrapper[2583]: E1030 20:38:43.854267    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30561fcc-3184-4116-b179-18941ff26a20-metrics-tls podName:30561fcc-3184-4116-b179-18941ff26a20 nodeName:}" failed. No retries permitted until 2026-10-30 20:38:45.85425226 +0000 UTC m=+34.030934798 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/30561fcc-3184-4116-b179-18941ff26a20-metrics-tls") pod "dns-default-fblmm" (UID: "30561fcc-3184-4116-b179-18941ff26a20") : secret "dns-default-metrics-tls" not found
Oct 30 20:38:45.871599 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:45.871513    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/30561fcc-3184-4116-b179-18941ff26a20-metrics-tls\") pod \"dns-default-fblmm\" (UID: \"30561fcc-3184-4116-b179-18941ff26a20\") " pod="openshift-dns/dns-default-fblmm"
Oct 30 20:38:45.872000 ip-10-0-131-125 kubenswrapper[2583]: E1030 20:38:45.871682    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 20:38:45.872000 ip-10-0-131-125 kubenswrapper[2583]: E1030 20:38:45.871753    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30561fcc-3184-4116-b179-18941ff26a20-metrics-tls podName:30561fcc-3184-4116-b179-18941ff26a20 nodeName:}" failed. No retries permitted until 2026-10-30 20:38:49.871737964 +0000 UTC m=+38.048420513 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/30561fcc-3184-4116-b179-18941ff26a20-metrics-tls") pod "dns-default-fblmm" (UID: "30561fcc-3184-4116-b179-18941ff26a20") : secret "dns-default-metrics-tls" not found
Oct 30 20:38:45.972740 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:45.972691    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a03cdbf5-91a2-41eb-8c1e-3c0f40da2632-original-pull-secret\") pod \"global-pull-secret-syncer-xkfsj\" (UID: \"a03cdbf5-91a2-41eb-8c1e-3c0f40da2632\") " pod="kube-system/global-pull-secret-syncer-xkfsj"
Oct 30 20:38:45.972740 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:45.972739    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6fdde40e-3330-4b84-a470-6a7e6a536dce-metrics-certs\") pod \"network-metrics-daemon-5bww9\" (UID: \"6fdde40e-3330-4b84-a470-6a7e6a536dce\") " pod="openshift-multus/network-metrics-daemon-5bww9"
Oct 30 20:38:45.972991 ip-10-0-131-125 kubenswrapper[2583]: E1030 20:38:45.972865    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 20:38:45.972991 ip-10-0-131-125 kubenswrapper[2583]: E1030 20:38:45.972934    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6fdde40e-3330-4b84-a470-6a7e6a536dce-metrics-certs podName:6fdde40e-3330-4b84-a470-6a7e6a536dce nodeName:}" failed. No retries permitted until 2026-10-30 20:39:17.972914624 +0000 UTC m=+66.149597166 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6fdde40e-3330-4b84-a470-6a7e6a536dce-metrics-certs") pod "network-metrics-daemon-5bww9" (UID: "6fdde40e-3330-4b84-a470-6a7e6a536dce") : secret "metrics-daemon-secret" not found
Oct 30 20:38:45.974980 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:45.974954    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a03cdbf5-91a2-41eb-8c1e-3c0f40da2632-original-pull-secret\") pod \"global-pull-secret-syncer-xkfsj\" (UID: \"a03cdbf5-91a2-41eb-8c1e-3c0f40da2632\") " pod="kube-system/global-pull-secret-syncer-xkfsj"
Oct 30 20:38:45.977342 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:45.977312    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xkfsj"
Oct 30 20:38:46.073719 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:46.073688    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqttz\" (UniqueName: \"kubernetes.io/projected/7957ff19-e0fc-42a7-967d-6e96e4a88094-kube-api-access-pqttz\") pod \"network-check-target-f4cch\" (UID: \"7957ff19-e0fc-42a7-967d-6e96e4a88094\") " pod="openshift-network-diagnostics/network-check-target-f4cch"
Oct 30 20:38:46.076179 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:46.076151    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqttz\" (UniqueName: \"kubernetes.io/projected/7957ff19-e0fc-42a7-967d-6e96e4a88094-kube-api-access-pqttz\") pod \"network-check-target-f4cch\" (UID: \"7957ff19-e0fc-42a7-967d-6e96e4a88094\") " pod="openshift-network-diagnostics/network-check-target-f4cch"
Oct 30 20:38:46.289899 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:46.289866    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-xkfsj"]
Oct 30 20:38:46.293259 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:46.293233    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda03cdbf5_91a2_41eb_8c1e_3c0f40da2632.slice/crio-2f51971ec9317807796205a24a53a6fc151c1eb896de31293a0d96ab562b1818 WatchSource:0}: Error finding container 2f51971ec9317807796205a24a53a6fc151c1eb896de31293a0d96ab562b1818: Status 404 returned error can't find the container with id 2f51971ec9317807796205a24a53a6fc151c1eb896de31293a0d96ab562b1818
Oct 30 20:38:46.320157 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:46.320136    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f4cch"
Oct 30 20:38:46.559530 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:46.559438    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-q2wtm" event={"ID":"747c15be-3782-4065-a6dc-73d9fe7a4b4f","Type":"ContainerStarted","Data":"8f36f5c49a9e8b54d130ac460a032eeb3ac720d5d6d4bce0e99d9d31e0f5cd7c"}
Oct 30 20:38:46.560551 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:46.560529    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-xkfsj" event={"ID":"a03cdbf5-91a2-41eb-8c1e-3c0f40da2632","Type":"ContainerStarted","Data":"2f51971ec9317807796205a24a53a6fc151c1eb896de31293a0d96ab562b1818"}
Oct 30 20:38:46.581155 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:46.579289    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-q2wtm" podStartSLOduration=1.749810991 podStartE2EDuration="4.579273458s" totalImagesPullingTime="2.829462467s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:38:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:38:46.578589526 +0000 UTC m=+34.755272073" watchObservedRunningTime="2026-10-30 20:38:46.579273458 +0000 UTC m=+34.755956005"
Oct 30 20:38:46.618887 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:46.618853    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-f4cch"]
Oct 30 20:38:46.621648 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:46.621619    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7957ff19_e0fc_42a7_967d_6e96e4a88094.slice/crio-fa8ba53d948c6e44b70f9ccb9e5acd0fa6720e0200e37cabbeee987f9b243239 WatchSource:0}: Error finding container fa8ba53d948c6e44b70f9ccb9e5acd0fa6720e0200e37cabbeee987f9b243239: Status 404 returned error can't find the container with id fa8ba53d948c6e44b70f9ccb9e5acd0fa6720e0200e37cabbeee987f9b243239
Oct 30 20:38:47.565024 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:47.564980    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-f4cch" event={"ID":"7957ff19-e0fc-42a7-967d-6e96e4a88094","Type":"ContainerStarted","Data":"fa8ba53d948c6e44b70f9ccb9e5acd0fa6720e0200e37cabbeee987f9b243239"}
Oct 30 20:38:48.195743 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:48.195703    2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-lkn2k"]
Oct 30 20:38:48.212002 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:48.211972    2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-4f444-dttkw"]
Oct 30 20:38:48.212445 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:48.212376    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-lkn2k"
Oct 30 20:38:48.222062 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:48.222035    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 20:38:48.222612 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:48.222471    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 20:38:48.222859 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:48.222710    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 20:38:48.222859 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:48.222769    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 20:38:48.223158 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:48.223137    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-6gb67\""
Oct 30 20:38:48.226017 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:48.225998    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-lkn2k"]
Oct 30 20:38:48.226107 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:48.226039    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-4f444-dttkw"]
Oct 30 20:38:48.226167 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:48.226136    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-4f444-dttkw"
Oct 30 20:38:48.319941 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:48.319904    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Oct 30 20:38:48.320126 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:48.320002    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-m94h2\""
Oct 30 20:38:48.324753 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:48.324731    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Oct 30 20:38:48.391680 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:48.391641    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9033c4e2-4586-4ec9-996b-21c412cfb8ba-service-ca-bundle\") pod \"periodic-gathering-4f444-dttkw\" (UID: \"9033c4e2-4586-4ec9-996b-21c412cfb8ba\") " pod="openshift-insights/periodic-gathering-4f444-dttkw"
Oct 30 20:38:48.391864 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:48.391685    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/451ecbeb-c188-4e49-8e4e-08af1051ae0f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lkn2k\" (UID: \"451ecbeb-c188-4e49-8e4e-08af1051ae0f\") " pod="openshift-insights/insights-runtime-extractor-lkn2k"
Oct 30 20:38:48.391864 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:48.391716    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26z6g\" (UniqueName: \"kubernetes.io/projected/451ecbeb-c188-4e49-8e4e-08af1051ae0f-kube-api-access-26z6g\") pod \"insights-runtime-extractor-lkn2k\" (UID: \"451ecbeb-c188-4e49-8e4e-08af1051ae0f\") " pod="openshift-insights/insights-runtime-extractor-lkn2k"
Oct 30 20:38:48.391864 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:48.391812    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/451ecbeb-c188-4e49-8e4e-08af1051ae0f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lkn2k\" (UID: \"451ecbeb-c188-4e49-8e4e-08af1051ae0f\") " pod="openshift-insights/insights-runtime-extractor-lkn2k"
Oct 30 20:38:48.392018 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:48.391882    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/451ecbeb-c188-4e49-8e4e-08af1051ae0f-crio-socket\") pod \"insights-runtime-extractor-lkn2k\" (UID: \"451ecbeb-c188-4e49-8e4e-08af1051ae0f\") " pod="openshift-insights/insights-runtime-extractor-lkn2k"
Oct 30 20:38:48.392018 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:48.391916    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/9033c4e2-4586-4ec9-996b-21c412cfb8ba-archives-path\") pod \"periodic-gathering-4f444-dttkw\" (UID: \"9033c4e2-4586-4ec9-996b-21c412cfb8ba\") " pod="openshift-insights/periodic-gathering-4f444-dttkw"
Oct 30 20:38:48.392018 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:48.391942    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/451ecbeb-c188-4e49-8e4e-08af1051ae0f-data-volume\") pod \"insights-runtime-extractor-lkn2k\" (UID: \"451ecbeb-c188-4e49-8e4e-08af1051ae0f\") " pod="openshift-insights/insights-runtime-extractor-lkn2k"
Oct 30 20:38:48.392018 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:48.391996    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lvnm\" (UniqueName: \"kubernetes.io/projected/9033c4e2-4586-4ec9-996b-21c412cfb8ba-kube-api-access-2lvnm\") pod \"periodic-gathering-4f444-dttkw\" (UID: \"9033c4e2-4586-4ec9-996b-21c412cfb8ba\") " pod="openshift-insights/periodic-gathering-4f444-dttkw"
Oct 30 20:38:48.392206 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:48.392046    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9033c4e2-4586-4ec9-996b-21c412cfb8ba-trusted-ca-bundle\") pod \"periodic-gathering-4f444-dttkw\" (UID: \"9033c4e2-4586-4ec9-996b-21c412cfb8ba\") " pod="openshift-insights/periodic-gathering-4f444-dttkw"
Oct 30 20:38:48.493345 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:48.493235    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26z6g\" (UniqueName: \"kubernetes.io/projected/451ecbeb-c188-4e49-8e4e-08af1051ae0f-kube-api-access-26z6g\") pod \"insights-runtime-extractor-lkn2k\" (UID: \"451ecbeb-c188-4e49-8e4e-08af1051ae0f\") " pod="openshift-insights/insights-runtime-extractor-lkn2k"
Oct 30 20:38:48.493345 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:48.493287    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/451ecbeb-c188-4e49-8e4e-08af1051ae0f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lkn2k\" (UID: \"451ecbeb-c188-4e49-8e4e-08af1051ae0f\") " pod="openshift-insights/insights-runtime-extractor-lkn2k"
Oct 30 20:38:48.493345 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:48.493341    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/451ecbeb-c188-4e49-8e4e-08af1051ae0f-crio-socket\") pod \"insights-runtime-extractor-lkn2k\" (UID: \"451ecbeb-c188-4e49-8e4e-08af1051ae0f\") " pod="openshift-insights/insights-runtime-extractor-lkn2k"
Oct 30 20:38:48.493610 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:48.493373    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/9033c4e2-4586-4ec9-996b-21c412cfb8ba-archives-path\") pod \"periodic-gathering-4f444-dttkw\" (UID: \"9033c4e2-4586-4ec9-996b-21c412cfb8ba\") " pod="openshift-insights/periodic-gathering-4f444-dttkw"
Oct 30 20:38:48.493610 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:48.493396    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/451ecbeb-c188-4e49-8e4e-08af1051ae0f-data-volume\") pod \"insights-runtime-extractor-lkn2k\" (UID: \"451ecbeb-c188-4e49-8e4e-08af1051ae0f\") " pod="openshift-insights/insights-runtime-extractor-lkn2k"
Oct 30 20:38:48.493610 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:48.493441    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lvnm\" (UniqueName: \"kubernetes.io/projected/9033c4e2-4586-4ec9-996b-21c412cfb8ba-kube-api-access-2lvnm\") pod \"periodic-gathering-4f444-dttkw\" (UID: \"9033c4e2-4586-4ec9-996b-21c412cfb8ba\") " pod="openshift-insights/periodic-gathering-4f444-dttkw"
Oct 30 20:38:48.493610 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:48.493490    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9033c4e2-4586-4ec9-996b-21c412cfb8ba-trusted-ca-bundle\") pod \"periodic-gathering-4f444-dttkw\" (UID: \"9033c4e2-4586-4ec9-996b-21c412cfb8ba\") " pod="openshift-insights/periodic-gathering-4f444-dttkw"
Oct 30 20:38:48.493610 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:48.493552    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9033c4e2-4586-4ec9-996b-21c412cfb8ba-service-ca-bundle\") pod \"periodic-gathering-4f444-dttkw\" (UID: \"9033c4e2-4586-4ec9-996b-21c412cfb8ba\") " pod="openshift-insights/periodic-gathering-4f444-dttkw"
Oct 30 20:38:48.493610 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:48.493576    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/451ecbeb-c188-4e49-8e4e-08af1051ae0f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lkn2k\" (UID: \"451ecbeb-c188-4e49-8e4e-08af1051ae0f\") " pod="openshift-insights/insights-runtime-extractor-lkn2k"
Oct 30 20:38:48.493844 ip-10-0-131-125 kubenswrapper[2583]: E1030 20:38:48.493670    2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 20:38:48.493844 ip-10-0-131-125 kubenswrapper[2583]: E1030 20:38:48.493741    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/451ecbeb-c188-4e49-8e4e-08af1051ae0f-kube-rbac-proxy-cm podName:451ecbeb-c188-4e49-8e4e-08af1051ae0f nodeName:}" failed. No retries permitted until 2026-10-30 20:38:48.993722451 +0000 UTC m=+37.170404997 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/451ecbeb-c188-4e49-8e4e-08af1051ae0f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lkn2k" (UID: "451ecbeb-c188-4e49-8e4e-08af1051ae0f") : configmap "kube-rbac-proxy" not found
Oct 30 20:38:48.494110 ip-10-0-131-125 kubenswrapper[2583]: E1030 20:38:48.494080    2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 20:38:48.494216 ip-10-0-131-125 kubenswrapper[2583]: E1030 20:38:48.494147    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/451ecbeb-c188-4e49-8e4e-08af1051ae0f-insights-runtime-extractor-tls podName:451ecbeb-c188-4e49-8e4e-08af1051ae0f nodeName:}" failed. No retries permitted until 2026-10-30 20:38:48.994129189 +0000 UTC m=+37.170811717 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/451ecbeb-c188-4e49-8e4e-08af1051ae0f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lkn2k" (UID: "451ecbeb-c188-4e49-8e4e-08af1051ae0f") : secret "insights-runtime-extractor-tls" not found
Oct 30 20:38:48.494358 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:48.494316    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/451ecbeb-c188-4e49-8e4e-08af1051ae0f-data-volume\") pod \"insights-runtime-extractor-lkn2k\" (UID: \"451ecbeb-c188-4e49-8e4e-08af1051ae0f\") " pod="openshift-insights/insights-runtime-extractor-lkn2k"
Oct 30 20:38:48.494450 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:48.494409    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/451ecbeb-c188-4e49-8e4e-08af1051ae0f-crio-socket\") pod \"insights-runtime-extractor-lkn2k\" (UID: \"451ecbeb-c188-4e49-8e4e-08af1051ae0f\") " pod="openshift-insights/insights-runtime-extractor-lkn2k"
Oct 30 20:38:48.494587 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:48.494563    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9033c4e2-4586-4ec9-996b-21c412cfb8ba-service-ca-bundle\") pod \"periodic-gathering-4f444-dttkw\" (UID: \"9033c4e2-4586-4ec9-996b-21c412cfb8ba\") " pod="openshift-insights/periodic-gathering-4f444-dttkw"
Oct 30 20:38:48.494664 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:48.494632    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/9033c4e2-4586-4ec9-996b-21c412cfb8ba-archives-path\") pod \"periodic-gathering-4f444-dttkw\" (UID: \"9033c4e2-4586-4ec9-996b-21c412cfb8ba\") " pod="openshift-insights/periodic-gathering-4f444-dttkw"
Oct 30 20:38:48.495038 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:48.495019    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9033c4e2-4586-4ec9-996b-21c412cfb8ba-trusted-ca-bundle\") pod \"periodic-gathering-4f444-dttkw\" (UID: \"9033c4e2-4586-4ec9-996b-21c412cfb8ba\") " pod="openshift-insights/periodic-gathering-4f444-dttkw"
Oct 30 20:38:48.513335 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:48.513263    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lvnm\" (UniqueName: \"kubernetes.io/projected/9033c4e2-4586-4ec9-996b-21c412cfb8ba-kube-api-access-2lvnm\") pod \"periodic-gathering-4f444-dttkw\" (UID: \"9033c4e2-4586-4ec9-996b-21c412cfb8ba\") " pod="openshift-insights/periodic-gathering-4f444-dttkw"
Oct 30 20:38:48.515601 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:48.515572    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26z6g\" (UniqueName: \"kubernetes.io/projected/451ecbeb-c188-4e49-8e4e-08af1051ae0f-kube-api-access-26z6g\") pod \"insights-runtime-extractor-lkn2k\" (UID: \"451ecbeb-c188-4e49-8e4e-08af1051ae0f\") " pod="openshift-insights/insights-runtime-extractor-lkn2k"
Oct 30 20:38:48.587035 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:48.587003    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-4f444-dttkw"
Oct 30 20:38:48.727335 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:48.727287    2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-f77w8"]
Oct 30 20:38:48.740651 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:48.740620    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-f77w8"]
Oct 30 20:38:48.740817 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:48.740759    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-f77w8"
Oct 30 20:38:48.751923 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:48.751899    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Oct 30 20:38:48.752050 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:48.751920    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Oct 30 20:38:48.752143 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:48.752125    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Oct 30 20:38:48.752209 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:48.752177    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Oct 30 20:38:48.752262 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:48.752251    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-rcwsg\""
Oct 30 20:38:48.796843 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:48.796804    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/1672e221-5127-44b8-bbc7-4d7b7ddc25c5-signing-key\") pod \"service-ca-5fffc97bdf-f77w8\" (UID: \"1672e221-5127-44b8-bbc7-4d7b7ddc25c5\") " pod="openshift-service-ca/service-ca-5fffc97bdf-f77w8"
Oct 30 20:38:48.796998 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:48.796941    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/1672e221-5127-44b8-bbc7-4d7b7ddc25c5-signing-cabundle\") pod \"service-ca-5fffc97bdf-f77w8\" (UID: \"1672e221-5127-44b8-bbc7-4d7b7ddc25c5\") " pod="openshift-service-ca/service-ca-5fffc97bdf-f77w8"
Oct 30 20:38:48.796998 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:48.796985    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8ggd\" (UniqueName: \"kubernetes.io/projected/1672e221-5127-44b8-bbc7-4d7b7ddc25c5-kube-api-access-r8ggd\") pod \"service-ca-5fffc97bdf-f77w8\" (UID: \"1672e221-5127-44b8-bbc7-4d7b7ddc25c5\") " pod="openshift-service-ca/service-ca-5fffc97bdf-f77w8"
Oct 30 20:38:48.817547 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:48.817506    2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-wqbfm"]
Oct 30 20:38:48.843809 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:48.843774    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-wqbfm"]
Oct 30 20:38:48.843964 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:48.843898    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-wqbfm"
Oct 30 20:38:48.859104 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:48.858899    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Oct 30 20:38:48.859104 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:48.858943    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Oct 30 20:38:48.859104 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:48.859066    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-8f8mm\""
Oct 30 20:38:48.897544 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:48.897512    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/1672e221-5127-44b8-bbc7-4d7b7ddc25c5-signing-key\") pod \"service-ca-5fffc97bdf-f77w8\" (UID: \"1672e221-5127-44b8-bbc7-4d7b7ddc25c5\") " pod="openshift-service-ca/service-ca-5fffc97bdf-f77w8"
Oct 30 20:38:48.897711 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:48.897627    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmvjx\" (UniqueName: \"kubernetes.io/projected/932cdb01-d506-4c20-9b4d-902da81e1f3c-kube-api-access-dmvjx\") pod \"migrator-66df5d496b-wqbfm\" (UID: \"932cdb01-d506-4c20-9b4d-902da81e1f3c\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-wqbfm"
Oct 30 20:38:48.897711 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:48.897671    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/1672e221-5127-44b8-bbc7-4d7b7ddc25c5-signing-cabundle\") pod \"service-ca-5fffc97bdf-f77w8\" (UID: \"1672e221-5127-44b8-bbc7-4d7b7ddc25c5\") " pod="openshift-service-ca/service-ca-5fffc97bdf-f77w8"
Oct 30 20:38:48.897711 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:48.897709    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8ggd\" (UniqueName: \"kubernetes.io/projected/1672e221-5127-44b8-bbc7-4d7b7ddc25c5-kube-api-access-r8ggd\") pod \"service-ca-5fffc97bdf-f77w8\" (UID: \"1672e221-5127-44b8-bbc7-4d7b7ddc25c5\") " pod="openshift-service-ca/service-ca-5fffc97bdf-f77w8"
Oct 30 20:38:48.898447 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:48.898418    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/1672e221-5127-44b8-bbc7-4d7b7ddc25c5-signing-cabundle\") pod \"service-ca-5fffc97bdf-f77w8\" (UID: \"1672e221-5127-44b8-bbc7-4d7b7ddc25c5\") " pod="openshift-service-ca/service-ca-5fffc97bdf-f77w8"
Oct 30 20:38:48.900669 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:48.900643    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/1672e221-5127-44b8-bbc7-4d7b7ddc25c5-signing-key\") pod \"service-ca-5fffc97bdf-f77w8\" (UID: \"1672e221-5127-44b8-bbc7-4d7b7ddc25c5\") " pod="openshift-service-ca/service-ca-5fffc97bdf-f77w8"
Oct 30 20:38:48.903853 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:48.903831    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-4f444-dttkw"]
Oct 30 20:38:48.907645 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:48.907615    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9033c4e2_4586_4ec9_996b_21c412cfb8ba.slice/crio-958c9bd7c6d62bcfd6effc523bd6f0835c3ba9f68387a4e1770cf0efdf7ae66b WatchSource:0}: Error finding container 958c9bd7c6d62bcfd6effc523bd6f0835c3ba9f68387a4e1770cf0efdf7ae66b: Status 404 returned error can't find the container with id 958c9bd7c6d62bcfd6effc523bd6f0835c3ba9f68387a4e1770cf0efdf7ae66b
Oct 30 20:38:48.911225 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:48.911204    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8ggd\" (UniqueName: \"kubernetes.io/projected/1672e221-5127-44b8-bbc7-4d7b7ddc25c5-kube-api-access-r8ggd\") pod \"service-ca-5fffc97bdf-f77w8\" (UID: \"1672e221-5127-44b8-bbc7-4d7b7ddc25c5\") " pod="openshift-service-ca/service-ca-5fffc97bdf-f77w8"
Oct 30 20:38:48.999157 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:48.999071    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/451ecbeb-c188-4e49-8e4e-08af1051ae0f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lkn2k\" (UID: \"451ecbeb-c188-4e49-8e4e-08af1051ae0f\") " pod="openshift-insights/insights-runtime-extractor-lkn2k"
Oct 30 20:38:48.999157 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:48.999126    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmvjx\" (UniqueName: \"kubernetes.io/projected/932cdb01-d506-4c20-9b4d-902da81e1f3c-kube-api-access-dmvjx\") pod \"migrator-66df5d496b-wqbfm\" (UID: \"932cdb01-d506-4c20-9b4d-902da81e1f3c\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-wqbfm"
Oct 30 20:38:48.999407 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:48.999156    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/451ecbeb-c188-4e49-8e4e-08af1051ae0f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lkn2k\" (UID: \"451ecbeb-c188-4e49-8e4e-08af1051ae0f\") " pod="openshift-insights/insights-runtime-extractor-lkn2k"
Oct 30 20:38:48.999407 ip-10-0-131-125 kubenswrapper[2583]: E1030 20:38:48.999170    2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 20:38:48.999407 ip-10-0-131-125 kubenswrapper[2583]: E1030 20:38:48.999287    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/451ecbeb-c188-4e49-8e4e-08af1051ae0f-kube-rbac-proxy-cm podName:451ecbeb-c188-4e49-8e4e-08af1051ae0f nodeName:}" failed. No retries permitted until 2026-10-30 20:38:49.999269405 +0000 UTC m=+38.175951952 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/451ecbeb-c188-4e49-8e4e-08af1051ae0f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lkn2k" (UID: "451ecbeb-c188-4e49-8e4e-08af1051ae0f") : configmap "kube-rbac-proxy" not found
Oct 30 20:38:48.999407 ip-10-0-131-125 kubenswrapper[2583]: E1030 20:38:48.999291    2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 20:38:48.999407 ip-10-0-131-125 kubenswrapper[2583]: E1030 20:38:48.999374    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/451ecbeb-c188-4e49-8e4e-08af1051ae0f-insights-runtime-extractor-tls podName:451ecbeb-c188-4e49-8e4e-08af1051ae0f nodeName:}" failed. No retries permitted until 2026-10-30 20:38:49.999357988 +0000 UTC m=+38.176040516 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/451ecbeb-c188-4e49-8e4e-08af1051ae0f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lkn2k" (UID: "451ecbeb-c188-4e49-8e4e-08af1051ae0f") : secret "insights-runtime-extractor-tls" not found
Oct 30 20:38:49.012901 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:49.012874    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmvjx\" (UniqueName: \"kubernetes.io/projected/932cdb01-d506-4c20-9b4d-902da81e1f3c-kube-api-access-dmvjx\") pod \"migrator-66df5d496b-wqbfm\" (UID: \"932cdb01-d506-4c20-9b4d-902da81e1f3c\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-wqbfm"
Oct 30 20:38:49.071905 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:49.071867    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-f77w8"
Oct 30 20:38:49.181465 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:49.181432    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-wqbfm"
Oct 30 20:38:49.374109 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:49.373955    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-f77w8"]
Oct 30 20:38:49.376831 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:49.376801    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1672e221_5127_44b8_bbc7_4d7b7ddc25c5.slice/crio-59229015d24ed2b2c8994862d50edd546044ceecc093c32fbf4815598cf2b7d2 WatchSource:0}: Error finding container 59229015d24ed2b2c8994862d50edd546044ceecc093c32fbf4815598cf2b7d2: Status 404 returned error can't find the container with id 59229015d24ed2b2c8994862d50edd546044ceecc093c32fbf4815598cf2b7d2
Oct 30 20:38:49.494110 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:49.494079    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-wqbfm"]
Oct 30 20:38:49.497648 ip-10-0-131-125 kubenswrapper[2583]: W1030 20:38:49.497589    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod932cdb01_d506_4c20_9b4d_902da81e1f3c.slice/crio-8c7f18802822bdf57f6cb5a40c00900280dea799439b726ea8050dc19c8bce0d WatchSource:0}: Error finding container 8c7f18802822bdf57f6cb5a40c00900280dea799439b726ea8050dc19c8bce0d: Status 404 returned error can't find the container with id 8c7f18802822bdf57f6cb5a40c00900280dea799439b726ea8050dc19c8bce0d
Oct 30 20:38:49.573045 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:49.572917    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-wqbfm" event={"ID":"932cdb01-d506-4c20-9b4d-902da81e1f3c","Type":"ContainerStarted","Data":"8c7f18802822bdf57f6cb5a40c00900280dea799439b726ea8050dc19c8bce0d"}
Oct 30 20:38:49.574457 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:49.574383    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-f77w8" event={"ID":"1672e221-5127-44b8-bbc7-4d7b7ddc25c5","Type":"ContainerStarted","Data":"59229015d24ed2b2c8994862d50edd546044ceecc093c32fbf4815598cf2b7d2"}
Oct 30 20:38:49.575843 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:49.575792    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-4f444-dttkw" event={"ID":"9033c4e2-4586-4ec9-996b-21c412cfb8ba","Type":"ContainerStarted","Data":"958c9bd7c6d62bcfd6effc523bd6f0835c3ba9f68387a4e1770cf0efdf7ae66b"}
Oct 30 20:38:49.905370 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:49.905256    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/30561fcc-3184-4116-b179-18941ff26a20-metrics-tls\") pod \"dns-default-fblmm\" (UID: \"30561fcc-3184-4116-b179-18941ff26a20\") " pod="openshift-dns/dns-default-fblmm"
Oct 30 20:38:49.905797 ip-10-0-131-125 kubenswrapper[2583]: E1030 20:38:49.905494    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 20:38:49.905797 ip-10-0-131-125 kubenswrapper[2583]: E1030 20:38:49.905572    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30561fcc-3184-4116-b179-18941ff26a20-metrics-tls podName:30561fcc-3184-4116-b179-18941ff26a20 nodeName:}" failed. No retries permitted until 2026-10-30 20:38:57.905551297 +0000 UTC m=+46.082233840 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/30561fcc-3184-4116-b179-18941ff26a20-metrics-tls") pod "dns-default-fblmm" (UID: "30561fcc-3184-4116-b179-18941ff26a20") : secret "dns-default-metrics-tls" not found
Oct 30 20:38:50.006162 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:50.006114    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/451ecbeb-c188-4e49-8e4e-08af1051ae0f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lkn2k\" (UID: \"451ecbeb-c188-4e49-8e4e-08af1051ae0f\") " pod="openshift-insights/insights-runtime-extractor-lkn2k"
Oct 30 20:38:50.006162 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:50.006166    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/451ecbeb-c188-4e49-8e4e-08af1051ae0f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lkn2k\" (UID: \"451ecbeb-c188-4e49-8e4e-08af1051ae0f\") " pod="openshift-insights/insights-runtime-extractor-lkn2k"
Oct 30 20:38:50.006395 ip-10-0-131-125 kubenswrapper[2583]: E1030 20:38:50.006307    2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 20:38:50.006395 ip-10-0-131-125 kubenswrapper[2583]: E1030 20:38:50.006382    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/451ecbeb-c188-4e49-8e4e-08af1051ae0f-insights-runtime-extractor-tls podName:451ecbeb-c188-4e49-8e4e-08af1051ae0f nodeName:}" failed. No retries permitted until 2026-10-30 20:38:52.006362907 +0000 UTC m=+40.183045455 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/451ecbeb-c188-4e49-8e4e-08af1051ae0f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lkn2k" (UID: "451ecbeb-c188-4e49-8e4e-08af1051ae0f") : secret "insights-runtime-extractor-tls" not found
Oct 30 20:38:50.006491 ip-10-0-131-125 kubenswrapper[2583]: E1030 20:38:50.006401    2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 20:38:50.006491 ip-10-0-131-125 kubenswrapper[2583]: E1030 20:38:50.006440    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/451ecbeb-c188-4e49-8e4e-08af1051ae0f-kube-rbac-proxy-cm podName:451ecbeb-c188-4e49-8e4e-08af1051ae0f nodeName:}" failed. No retries permitted until 2026-10-30 20:38:52.006429679 +0000 UTC m=+40.183112210 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/451ecbeb-c188-4e49-8e4e-08af1051ae0f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lkn2k" (UID: "451ecbeb-c188-4e49-8e4e-08af1051ae0f") : configmap "kube-rbac-proxy" not found
Oct 30 20:38:51.584201 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:51.584113    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-f4cch" event={"ID":"7957ff19-e0fc-42a7-967d-6e96e4a88094","Type":"ContainerStarted","Data":"b09cc255020b52b49bdf03c9f80ba3b418dd61aab04502313ab3b59bf6524f7a"}
Oct 30 20:38:51.584636 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:51.584294    2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-f4cch"
Oct 30 20:38:51.602892 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:51.602833    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-f4cch" podStartSLOduration=35.674803612 podStartE2EDuration="39.602814068s" totalImagesPullingTime="3.928010456s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:38:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:38:51.601771668 +0000 UTC m=+39.778454216" watchObservedRunningTime="2026-10-30 20:38:51.602814068 +0000 UTC m=+39.779496619"
Oct 30 20:38:52.023700 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:52.023659    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/451ecbeb-c188-4e49-8e4e-08af1051ae0f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lkn2k\" (UID: \"451ecbeb-c188-4e49-8e4e-08af1051ae0f\") " pod="openshift-insights/insights-runtime-extractor-lkn2k"
Oct 30 20:38:52.023875 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:52.023715    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/451ecbeb-c188-4e49-8e4e-08af1051ae0f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lkn2k\" (UID: \"451ecbeb-c188-4e49-8e4e-08af1051ae0f\") " pod="openshift-insights/insights-runtime-extractor-lkn2k"
Oct 30 20:38:52.023947 ip-10-0-131-125 kubenswrapper[2583]: E1030 20:38:52.023877    2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 20:38:52.023947 ip-10-0-131-125 kubenswrapper[2583]: E1030 20:38:52.023944    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/451ecbeb-c188-4e49-8e4e-08af1051ae0f-insights-runtime-extractor-tls podName:451ecbeb-c188-4e49-8e4e-08af1051ae0f nodeName:}" failed. No retries permitted until 2026-10-30 20:38:56.023924281 +0000 UTC m=+44.200606826 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/451ecbeb-c188-4e49-8e4e-08af1051ae0f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lkn2k" (UID: "451ecbeb-c188-4e49-8e4e-08af1051ae0f") : secret "insights-runtime-extractor-tls" not found
Oct 30 20:38:52.024306 ip-10-0-131-125 kubenswrapper[2583]: E1030 20:38:52.024284    2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 20:38:52.024442 ip-10-0-131-125 kubenswrapper[2583]: E1030 20:38:52.024353    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/451ecbeb-c188-4e49-8e4e-08af1051ae0f-kube-rbac-proxy-cm podName:451ecbeb-c188-4e49-8e4e-08af1051ae0f nodeName:}" failed. No retries permitted until 2026-10-30 20:38:56.024338112 +0000 UTC m=+44.201020651 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/451ecbeb-c188-4e49-8e4e-08af1051ae0f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lkn2k" (UID: "451ecbeb-c188-4e49-8e4e-08af1051ae0f") : configmap "kube-rbac-proxy" not found
Oct 30 20:38:52.588516 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:52.588428    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-xkfsj" event={"ID":"a03cdbf5-91a2-41eb-8c1e-3c0f40da2632","Type":"ContainerStarted","Data":"abd9119189b5fa4a04f1999815f0c40b8af4bc711a7c1d3da93ddf8a60b7587c"}
Oct 30 20:38:52.589979 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:52.589950    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-wqbfm" event={"ID":"932cdb01-d506-4c20-9b4d-902da81e1f3c","Type":"ContainerStarted","Data":"c9b2cd9e59f7a6e1ce324fd31d9a7b56f02439cfd0ad7494e723b781a6dd0cb3"}
Oct 30 20:38:52.592805 ip-10-0-131-125 kubenswrapper[2583]: I1030 20:38:52.592779    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-4f444-dttkw" event={"ID":"9033c4e2-4586-4ec9-996b-21c412cfb8ba","Type":"ContainerStarted","Data":"84f75d512bda86246800d98992feae6f917091e0b59369751f878ee85d0ae742"}
Oct 30 20:38:52.596437 ip-10-0-131-125 systemd[1]: Stopping Kubernetes Kubelet...
Oct 30 20:38:52.604859 ip-10-0-131-125 systemd[1]: kubelet.service: Deactivated successfully.
Oct 30 20:38:52.605026 ip-10-0-131-125 systemd[1]: Stopped Kubernetes Kubelet.
Oct 30 20:38:52.605076 ip-10-0-131-125 systemd[1]: kubelet.service: Consumed 2.428s CPU time.
Oct 30 20:38:52.613836 ip-10-0-131-125 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 20:38:52.708266 ip-10-0-131-125 kubenswrapper[4521]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 20:38:52.708266 ip-10-0-131-125 kubenswrapper[4521]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 20:38:52.708266 ip-10-0-131-125 kubenswrapper[4521]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 20:38:52.708266 ip-10-0-131-125 kubenswrapper[4521]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 20:38:52.711215 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.711120    4521 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 20:38:52.711215 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.711212    4521 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 20:38:52.711215 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.711217    4521 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 20:38:52.711215 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.711220    4521 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 20:38:52.711408 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.711224    4521 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 20:38:52.711408 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.711227    4521 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 20:38:52.711408 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.711229    4521 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 20:38:52.711408 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.711232    4521 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 20:38:52.711408 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.711235    4521 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 20:38:52.711408 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.711238    4521 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 20:38:52.711408 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.711240    4521 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 20:38:52.711408 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.711245    4521 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 20:38:52.711408 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.711249    4521 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 20:38:52.711408 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.711253    4521 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 20:38:52.711408 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.711257    4521 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 20:38:52.711408 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.711260    4521 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 20:38:52.711408 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.711263    4521 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 20:38:52.711408 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.711265    4521 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 20:38:52.711408 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.711268    4521 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 20:38:52.711408 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.711271    4521 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 20:38:52.711408 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.711274    4521 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 20:38:52.711408 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.711277    4521 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 20:38:52.711857 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.711280    4521 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 20:38:52.711857 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.711282    4521 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 20:38:52.711857 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.711285    4521 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 20:38:52.711857 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.711288    4521 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 20:38:52.711857 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.711291    4521 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 20:38:52.711857 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.711293    4521 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 20:38:52.711857 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.711296    4521 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 20:38:52.711857 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.711299    4521 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 20:38:52.711857 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.711301    4521 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 20:38:52.711857 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.711304    4521 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 20:38:52.711857 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.711307    4521 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 20:38:52.711857 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.711310    4521 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 20:38:52.711857 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.711312    4521 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 20:38:52.711857 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.711315    4521 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 20:38:52.711857 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.711318    4521 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 20:38:52.711857 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.711334    4521 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 20:38:52.711857 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.711337    4521 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 20:38:52.711857 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.711340    4521 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 20:38:52.711857 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.711343    4521 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 20:38:52.712316 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.711347    4521 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 20:38:52.712316 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.711351    4521 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 20:38:52.712316 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.711354    4521 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 20:38:52.712316 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.711357    4521 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 20:38:52.712316 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.711360    4521 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 20:38:52.712316 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.711363    4521 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 20:38:52.712316 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.711366    4521 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 20:38:52.712316 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.711369    4521 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 20:38:52.712316 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.711371    4521 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 20:38:52.712316 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.711375    4521 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 20:38:52.712316 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.711378    4521 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 20:38:52.712316 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.711381    4521 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 20:38:52.712316 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.711385    4521 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 20:38:52.712316 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.711387    4521 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 20:38:52.712316 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.711390    4521 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 20:38:52.712316 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.711392    4521 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 20:38:52.712316 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.711395    4521 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 20:38:52.712316 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.711398    4521 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 20:38:52.712316 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.711401    4521 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 20:38:52.712316 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.711403    4521 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 20:38:52.712801 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.711407    4521 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 20:38:52.712801 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.711409    4521 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 20:38:52.712801 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.711412    4521 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 20:38:52.712801 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.711415    4521 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 20:38:52.712801 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.711417    4521 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 20:38:52.712801 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.711421    4521 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 20:38:52.712801 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.711423    4521 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 20:38:52.712801 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.711426    4521 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 20:38:52.712801 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.711429    4521 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 20:38:52.712801 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.711432    4521 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 20:38:52.712801 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.711434    4521 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 20:38:52.712801 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.711437    4521 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 20:38:52.712801 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.711439    4521 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 20:38:52.712801 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.711442    4521 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 20:38:52.712801 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.711445    4521 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 20:38:52.712801 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.711447    4521 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 20:38:52.712801 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.711450    4521 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 20:38:52.712801 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.711453    4521 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 20:38:52.712801 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.711456    4521 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 20:38:52.713271 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.711459    4521 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 20:38:52.713271 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.711462    4521 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 20:38:52.713271 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.711464    4521 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 20:38:52.713271 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.711467    4521 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 20:38:52.713271 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.711470    4521 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 20:38:52.713271 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.711473    4521 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 20:38:52.713271 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.711477    4521 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 20:38:52.713271 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.711479    4521 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 20:38:52.713271 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.711482    4521 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 20:38:52.713271 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.711484    4521 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 20:38:52.713271 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.711487    4521 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 20:38:52.713271 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.711489    4521 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 20:38:52.713271 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.711492    4521 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 20:38:52.713271 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.711495    4521 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 20:38:52.713271 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.711497    4521 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 20:38:52.713271 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.711500    4521 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 20:38:52.713271 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.711502    4521 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 20:38:52.713271 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.711505    4521 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 20:38:52.713271 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.711508    4521 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 20:38:52.713753 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.711510    4521 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 20:38:52.713753 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.711516    4521 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 20:38:52.713753 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.711519    4521 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 20:38:52.713753 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.711521    4521 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 20:38:52.713753 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.711524    4521 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 20:38:52.713753 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.711526    4521 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 20:38:52.713753 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.711529    4521 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 20:38:52.713753 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.712291    4521 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 20:38:52.713753 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.712297    4521 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 20:38:52.713753 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.712300    4521 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 20:38:52.713753 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.712303    4521 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 20:38:52.713753 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.712306    4521 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 20:38:52.713753 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.712309    4521 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 20:38:52.713753 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.712312    4521 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 20:38:52.713753 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.712315    4521 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 20:38:52.713753 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.712317    4521 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 20:38:52.713753 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.712335    4521 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 20:38:52.713753 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.712339    4521 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 20:38:52.713753 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.712342    4521 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 20:38:52.713753 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.712344    4521 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 20:38:52.714273 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.712347    4521 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 20:38:52.714273 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.712350    4521 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 20:38:52.714273 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.712352    4521 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 20:38:52.714273 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.712355    4521 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 20:38:52.714273 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.712358    4521 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 20:38:52.714273 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.712361    4521 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 20:38:52.714273 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.712364    4521 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 20:38:52.714273 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.712366    4521 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 20:38:52.714273 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.712369    4521 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 20:38:52.714273 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.712372    4521 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 20:38:52.714273 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.712375    4521 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 20:38:52.714273 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.712377    4521 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 20:38:52.714273 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.712380    4521 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 20:38:52.714273 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.712385    4521 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 20:38:52.714273 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.712388    4521 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 20:38:52.714273 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.712393    4521 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 20:38:52.714273 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.712397    4521 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 20:38:52.714273 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.712400    4521 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 20:38:52.714723 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.712403    4521 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 20:38:52.714723 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.712405    4521 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 20:38:52.714723 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.712408    4521 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 20:38:52.714723 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.712410    4521 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 20:38:52.714723 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.712413    4521 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 20:38:52.714723 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.712415    4521 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 20:38:52.714723 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.712418    4521 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 20:38:52.714723 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.712421    4521 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 20:38:52.714723 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.712424    4521 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 20:38:52.714723 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.712427    4521 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 20:38:52.714723 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.712429    4521 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 20:38:52.714723 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.712432    4521 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 20:38:52.714723 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.712436    4521 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 20:38:52.714723 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.712439    4521 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 20:38:52.714723 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.712441    4521 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 20:38:52.714723 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.712444    4521 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 20:38:52.714723 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.712447    4521 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 20:38:52.714723 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.712449    4521 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 20:38:52.714723 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.712452    4521 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 20:38:52.714723 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.712454    4521 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 20:38:52.715210 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.712457    4521 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 20:38:52.715210 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.712460    4521 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 20:38:52.715210 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.712463    4521 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 20:38:52.715210 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.712466    4521 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 20:38:52.715210 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.712468    4521 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 20:38:52.715210 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.712471    4521 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 20:38:52.715210 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.712473    4521 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 20:38:52.715210 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.712477    4521 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 20:38:52.715210 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.712479    4521 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 20:38:52.715210 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.712482    4521 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 20:38:52.715210 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.712485    4521 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 20:38:52.715210 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.712488    4521 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 20:38:52.715210 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.712490    4521 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 20:38:52.715210 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.712493    4521 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 20:38:52.715210 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.712496    4521 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 20:38:52.715210 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.712499    4521 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 20:38:52.715210 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.712502    4521 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 20:38:52.715210 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.712504    4521 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 20:38:52.715210 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.712507    4521 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 20:38:52.715210 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.712510    4521 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 20:38:52.715713 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.712512    4521 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 20:38:52.715713 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.712515    4521 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 20:38:52.715713 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.712517    4521 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 20:38:52.715713 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.712519    4521 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 20:38:52.715713 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.712522    4521 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 20:38:52.715713 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.712526    4521 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 20:38:52.715713 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.712529    4521 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 20:38:52.715713 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.712532    4521 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 20:38:52.715713 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.712534    4521 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 20:38:52.715713 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.712537    4521 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 20:38:52.715713 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.712539    4521 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 20:38:52.715713 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.712541    4521 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 20:38:52.715713 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.712544    4521 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 20:38:52.715713 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.712547    4521 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 20:38:52.715713 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.712549    4521 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 20:38:52.715713 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.712552    4521 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 20:38:52.715713 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.712554    4521 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 20:38:52.715713 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.712557    4521 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 20:38:52.715713 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.712559    4521 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 20:38:52.715713 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.712562    4521 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 20:38:52.716208 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.712565    4521 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 20:38:52.716208 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.712567    4521 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 20:38:52.716208 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.712570    4521 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 20:38:52.716208 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.712573    4521 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 20:38:52.716208 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.712576    4521 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 20:38:52.716208 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.712579    4521 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 20:38:52.716208 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.712581    4521 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 20:38:52.716208 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.712584    4521 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 20:38:52.716208 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.712586    4521 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 20:38:52.716208 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.712591    4521 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 20:38:52.716208 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.712593    4521 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 20:38:52.716208 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.712596    4521 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 20:38:52.716208 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.712599    4521 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 20:38:52.716208 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.712601    4521 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 20:38:52.716208 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.712604    4521 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 20:38:52.716208 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.713109    4521 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 20:38:52.716208 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.713114    4521 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 20:38:52.716208 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.713118    4521 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 20:38:52.716208 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.713120    4521 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 20:38:52.716673 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.713123    4521 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 20:38:52.716673 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.713126    4521 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 20:38:52.716673 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.713129    4521 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 20:38:52.716673 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.713132    4521 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 20:38:52.716673 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.713135    4521 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 20:38:52.716673 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.713137    4521 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 20:38:52.716673 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.713140    4521 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 20:38:52.716673 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.713143    4521 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 20:38:52.716673 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.713146    4521 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 20:38:52.716673 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.713150    4521 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 20:38:52.716673 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.713154    4521 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 20:38:52.716673 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.713157    4521 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 20:38:52.716673 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.713161    4521 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 20:38:52.716673 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.713163    4521 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 20:38:52.716673 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.713166    4521 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 20:38:52.716673 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.713169    4521 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 20:38:52.716673 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.713172    4521 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 20:38:52.716673 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.713175    4521 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 20:38:52.716673 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.713178    4521 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 20:38:52.717142 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.713180    4521 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 20:38:52.717142 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.713183    4521 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 20:38:52.717142 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.713185    4521 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 20:38:52.717142 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.713188    4521 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 20:38:52.717142 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.713191    4521 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 20:38:52.717142 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.713193    4521 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 20:38:52.717142 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.713196    4521 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 20:38:52.717142 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.713198    4521 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 20:38:52.717142 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.713201    4521 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 20:38:52.717142 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.713204    4521 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 20:38:52.717142 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.713206    4521 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 20:38:52.717142 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.713209    4521 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 20:38:52.717142 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.713212    4521 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 20:38:52.717142 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.713214    4521 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 20:38:52.717142 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.713217    4521 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 20:38:52.717142 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.713219    4521 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 20:38:52.717142 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.713222    4521 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 20:38:52.717142 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.713225    4521 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 20:38:52.717142 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.713227    4521 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 20:38:52.717142 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.713230    4521 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 20:38:52.717637 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.713232    4521 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 20:38:52.717637 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.713235    4521 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 20:38:52.717637 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.713237    4521 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 20:38:52.717637 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.713240    4521 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 20:38:52.717637 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.713244    4521 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 20:38:52.717637 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.713247    4521 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 20:38:52.717637 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.713250    4521 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 20:38:52.717637 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.713253    4521 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 20:38:52.717637 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.713255    4521 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 20:38:52.717637 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.713258    4521 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 20:38:52.717637 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.713261    4521 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 20:38:52.717637 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.713263    4521 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 20:38:52.717637 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.713266    4521 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 20:38:52.717637 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.713268    4521 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 20:38:52.717637 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.713271    4521 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 20:38:52.717637 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.713274    4521 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 20:38:52.717637 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.713277    4521 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 20:38:52.717637 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.713279    4521 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 20:38:52.717637 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.713282    4521 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 20:38:52.718135 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.713285    4521 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 20:38:52.718135 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.713287    4521 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 20:38:52.718135 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.713290    4521 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 20:38:52.718135 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.713292    4521 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 20:38:52.718135 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.713295    4521 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 20:38:52.718135 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.713297    4521 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 20:38:52.718135 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.713301    4521 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 20:38:52.718135 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.713304    4521 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 20:38:52.718135 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.713306    4521 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 20:38:52.718135 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.713309    4521 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 20:38:52.718135 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.713311    4521 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 20:38:52.718135 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.713314    4521 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 20:38:52.718135 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.713317    4521 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 20:38:52.718135 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.713333    4521 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 20:38:52.718135 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.713336    4521 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 20:38:52.718135 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.713338    4521 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 20:38:52.718135 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.713341    4521 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 20:38:52.718135 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.713345    4521 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 20:38:52.718135 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.713349    4521 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 20:38:52.718135 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.713352    4521 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 20:38:52.718757 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.713355    4521 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 20:38:52.718757 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.713358    4521 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 20:38:52.718757 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.713361    4521 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 20:38:52.718757 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.713363    4521 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 20:38:52.718757 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.713366    4521 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 20:38:52.718757 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.713368    4521 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 20:38:52.718757 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.713371    4521 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 20:38:52.718757 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.713374    4521 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 20:38:52.718757 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.713376    4521 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 20:38:52.718757 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.713379    4521 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 20:38:52.718757 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.713381    4521 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 20:38:52.718757 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.713384    4521 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 20:38:52.718757 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.713387    4521 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 20:38:52.718757 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.713389    4521 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 20:38:52.718757 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.713392    4521 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 20:38:52.718757 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.713395    4521 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 20:38:52.718757 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.713397    4521 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 20:38:52.718757 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.713400    4521 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 20:38:52.718757 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.713403    4521 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 20:38:52.719392 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.713406    4521 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 20:38:52.719392 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.713409    4521 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 20:38:52.719392 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.713413    4521 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 20:38:52.719392 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.713417    4521 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 20:38:52.719392 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.713420    4521 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 20:38:52.719392 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.713512    4521 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 20:38:52.719392 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.713520    4521 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 20:38:52.719392 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.713526    4521 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 20:38:52.719392 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.713532    4521 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 20:38:52.719392 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.713536    4521 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 20:38:52.719392 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.713542    4521 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 20:38:52.719392 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.713546    4521 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 20:38:52.719392 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.713551    4521 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 20:38:52.719392 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.713554    4521 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 20:38:52.719392 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.713558    4521 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 20:38:52.719392 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.713562    4521 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 20:38:52.719392 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.713565    4521 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 20:38:52.719392 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.713569    4521 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 20:38:52.719392 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.713572    4521 flags.go:64] FLAG: --cgroup-root=""
Oct 30 20:38:52.719392 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.713575    4521 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 20:38:52.719392 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.713578    4521 flags.go:64] FLAG: --client-ca-file=""
Oct 30 20:38:52.719392 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.713582    4521 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 20:38:52.719921 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.713585    4521 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 20:38:52.719921 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.713589    4521 flags.go:64] FLAG: --cluster-domain=""
Oct 30 20:38:52.719921 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.713592    4521 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 20:38:52.719921 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.713596    4521 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 20:38:52.719921 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.713599    4521 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 20:38:52.719921 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.713602    4521 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 20:38:52.719921 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.713611    4521 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 20:38:52.719921 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.713614    4521 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 20:38:52.719921 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.713617    4521 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 20:38:52.719921 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.713621    4521 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 20:38:52.719921 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.713624    4521 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 20:38:52.719921 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.713628    4521 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 20:38:52.719921 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.713631    4521 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 20:38:52.719921 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.713634    4521 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 20:38:52.719921 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.713637    4521 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 20:38:52.719921 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.713641    4521 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 20:38:52.719921 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.713644    4521 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 20:38:52.719921 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.713648    4521 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 20:38:52.719921 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.713650    4521 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 20:38:52.719921 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.713654    4521 flags.go:64] FLAG: --enable-server="true"
Oct 30 20:38:52.719921 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.713658    4521 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 20:38:52.719921 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.713666    4521 flags.go:64] FLAG: --event-burst="100"
Oct 30 20:38:52.719921 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.713669    4521 flags.go:64] FLAG: --event-qps="50"
Oct 30 20:38:52.719921 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.713673    4521 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 20:38:52.720574 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.713676    4521 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 20:38:52.720574 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.713679    4521 flags.go:64] FLAG: --eviction-hard=""
Oct 30 20:38:52.720574 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.713684    4521 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 20:38:52.720574 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.713687    4521 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 20:38:52.720574 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.713690    4521 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 20:38:52.720574 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.713693    4521 flags.go:64] FLAG: --eviction-soft=""
Oct 30 20:38:52.720574 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.713696    4521 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 20:38:52.720574 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.713700    4521 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 20:38:52.720574 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.713703    4521 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 20:38:52.720574 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.713705    4521 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 20:38:52.720574 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.713708    4521 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 20:38:52.720574 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.713712    4521 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 20:38:52.720574 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.713715    4521 flags.go:64] FLAG: --feature-gates=""
Oct 30 20:38:52.720574 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.713719    4521 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 20:38:52.720574 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.713722    4521 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 20:38:52.720574 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.713725    4521 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 20:38:52.720574 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.713729    4521 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 20:38:52.720574 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.713732    4521 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 20:38:52.720574 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.713735    4521 flags.go:64] FLAG: --help="false"
Oct 30 20:38:52.720574 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.713738    4521 flags.go:64] FLAG: --hostname-override="ip-10-0-131-125.us-east-2.compute.internal"
Oct 30 20:38:52.720574 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.713742    4521 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 20:38:52.720574 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.713745    4521 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 20:38:52.720574 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.713748    4521 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 20:38:52.721134 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.713752    4521 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 20:38:52.721134 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.713756    4521 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 20:38:52.721134 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.713759    4521 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 20:38:52.721134 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.713762    4521 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 20:38:52.721134 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.713765    4521 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 20:38:52.721134 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.713769    4521 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 20:38:52.721134 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.713772    4521 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 20:38:52.721134 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.713775    4521 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 20:38:52.721134 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.713778    4521 flags.go:64] FLAG: --kube-reserved=""
Oct 30 20:38:52.721134 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.713781    4521 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 20:38:52.721134 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.713784    4521 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 20:38:52.721134 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.713787    4521 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 20:38:52.721134 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.713790    4521 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 20:38:52.721134 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.713793    4521 flags.go:64] FLAG: --lock-file=""
Oct 30 20:38:52.721134 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.713796    4521 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 20:38:52.721134 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.713799    4521 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 20:38:52.721134 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.713802    4521 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 20:38:52.721134 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.713808    4521 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 20:38:52.721134 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.713811    4521 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 20:38:52.721134 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.713814    4521 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 20:38:52.721134 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.713817    4521 flags.go:64] FLAG: --logging-format="text"
Oct 30 20:38:52.721134 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.713820    4521 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 20:38:52.721134 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.713823    4521 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 20:38:52.721134 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.713826    4521 flags.go:64] FLAG: --manifest-url=""
Oct 30 20:38:52.721759 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.713829    4521 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 20:38:52.721759 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.713833    4521 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 20:38:52.721759 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.713836    4521 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 20:38:52.721759 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.713841    4521 flags.go:64] FLAG: --max-pods="110"
Oct 30 20:38:52.721759 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.713844    4521 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 20:38:52.721759 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.713848    4521 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 20:38:52.721759 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.713851    4521 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 20:38:52.721759 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.713855    4521 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 20:38:52.721759 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.713858    4521 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 20:38:52.721759 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.713861    4521 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 20:38:52.721759 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.713864    4521 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 20:38:52.721759 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.713871    4521 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 20:38:52.721759 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.713874    4521 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 20:38:52.721759 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.713879    4521 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 20:38:52.721759 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.713882    4521 flags.go:64] FLAG: --pod-cidr=""
Oct 30 20:38:52.721759 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.713885    4521 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 20:38:52.721759 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.713888    4521 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 20:38:52.721759 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.713891    4521 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 20:38:52.721759 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.713894    4521 flags.go:64] FLAG: --port="10250"
Oct 30 20:38:52.721759 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.713897    4521 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 20:38:52.721759 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.713900    4521 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-0775184a58f4035a6"
Oct 30 20:38:52.721759 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.713903    4521 flags.go:64] FLAG: --qos-reserved=""
Oct 30 20:38:52.721759 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.713906    4521 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 20:38:52.721759 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.713909    4521 flags.go:64] FLAG: --register-node="true"
Oct 30 20:38:52.721759 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.713912    4521 flags.go:64] FLAG: --register-with-taints=""
Oct 30 20:38:52.721759 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.713916    4521 flags.go:64] FLAG: --registry-burst="10"
Oct 30 20:38:52.722393 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.713920    4521 flags.go:64] FLAG: --registry-qps="5"
Oct 30 20:38:52.722393 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.713922    4521 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 20:38:52.722393 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.713926    4521 flags.go:64] FLAG: --reserved-memory=""
Oct 30 20:38:52.722393 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.713930    4521 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 20:38:52.722393 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.713933    4521 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 20:38:52.722393 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.713936    4521 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 20:38:52.722393 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.713939    4521 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 20:38:52.722393 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.713943    4521 flags.go:64] FLAG: --runonce="false"
Oct 30 20:38:52.722393 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.713946    4521 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 20:38:52.722393 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.713949    4521 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 20:38:52.722393 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.713953    4521 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 20:38:52.722393 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.713956    4521 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 20:38:52.722393 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.713959    4521 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 20:38:52.722393 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.713962    4521 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 20:38:52.722393 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.713965    4521 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 20:38:52.722393 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.713968    4521 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 20:38:52.722393 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.713971    4521 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 20:38:52.722393 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.713974    4521 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 20:38:52.722393 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.713977    4521 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 20:38:52.722393 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.713982    4521 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 20:38:52.722393 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.713985    4521 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 20:38:52.722393 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.713988    4521 flags.go:64] FLAG: --system-cgroups=""
Oct 30 20:38:52.722393 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.713991    4521 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 20:38:52.722393 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.713996    4521 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 20:38:52.722393 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.713999    4521 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 20:38:52.723050 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.714002    4521 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 20:38:52.723050 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.714006    4521 flags.go:64] FLAG: --tls-min-version=""
Oct 30 20:38:52.723050 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.714009    4521 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 20:38:52.723050 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.714012    4521 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 20:38:52.723050 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.714015    4521 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 20:38:52.723050 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.714017    4521 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 20:38:52.723050 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.714021    4521 flags.go:64] FLAG: --v="2"
Oct 30 20:38:52.723050 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.714025    4521 flags.go:64] FLAG: --version="false"
Oct 30 20:38:52.723050 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.714029    4521 flags.go:64] FLAG: --vmodule=""
Oct 30 20:38:52.723050 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.714033    4521 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 20:38:52.723050 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.714040    4521 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 20:38:52.723050 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714175    4521 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 20:38:52.723050 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714179    4521 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 20:38:52.723050 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714181    4521 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 20:38:52.723050 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714185    4521 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 20:38:52.723050 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714188    4521 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 20:38:52.723050 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714191    4521 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 20:38:52.723050 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714194    4521 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 20:38:52.723050 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714196    4521 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 20:38:52.723050 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714199    4521 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 20:38:52.723050 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714201    4521 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 20:38:52.723050 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714204    4521 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 20:38:52.723911 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714208    4521 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 20:38:52.723911 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714210    4521 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 20:38:52.723911 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714213    4521 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 20:38:52.723911 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714215    4521 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 20:38:52.723911 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714219    4521 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 20:38:52.723911 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714222    4521 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 20:38:52.723911 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714225    4521 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 20:38:52.723911 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714227    4521 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 20:38:52.723911 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714230    4521 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 20:38:52.723911 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714232    4521 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 20:38:52.723911 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714235    4521 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 20:38:52.723911 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714237    4521 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 20:38:52.723911 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714240    4521 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 20:38:52.723911 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714243    4521 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 20:38:52.723911 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714245    4521 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 20:38:52.723911 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714248    4521 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 20:38:52.723911 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714250    4521 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 20:38:52.723911 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714253    4521 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 20:38:52.723911 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714256    4521 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 20:38:52.724615 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714258    4521 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 20:38:52.724615 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714261    4521 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 20:38:52.724615 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714263    4521 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 20:38:52.724615 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714266    4521 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 20:38:52.724615 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714268    4521 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 20:38:52.724615 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714272    4521 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 20:38:52.724615 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714275    4521 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 20:38:52.724615 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714278    4521 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 20:38:52.724615 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714280    4521 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 20:38:52.724615 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714283    4521 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 20:38:52.724615 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714286    4521 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 20:38:52.724615 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714288    4521 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 20:38:52.724615 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714291    4521 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 20:38:52.724615 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714294    4521 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 20:38:52.724615 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714297    4521 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 20:38:52.724615 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714300    4521 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 20:38:52.724615 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714302    4521 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 20:38:52.724615 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714306    4521 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 20:38:52.724615 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714310    4521 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 20:38:52.725394 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714313    4521 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 20:38:52.725394 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714316    4521 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 20:38:52.725394 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714332    4521 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 20:38:52.725394 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714335    4521 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 20:38:52.725394 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714338    4521 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 20:38:52.725394 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714340    4521 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 20:38:52.725394 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714343    4521 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 20:38:52.725394 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714345    4521 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 20:38:52.725394 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714348    4521 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 20:38:52.725394 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714351    4521 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 20:38:52.725394 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714354    4521 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 20:38:52.725394 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714356    4521 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 20:38:52.725394 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714359    4521 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 20:38:52.725394 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714362    4521 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 20:38:52.725394 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714364    4521 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 20:38:52.725394 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714367    4521 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 20:38:52.725394 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714370    4521 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 20:38:52.725394 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714373    4521 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 20:38:52.725394 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714376    4521 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 20:38:52.725394 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714379    4521 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 20:38:52.726006 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714381    4521 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 20:38:52.726006 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714384    4521 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 20:38:52.726006 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714387    4521 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 20:38:52.726006 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714389    4521 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 20:38:52.726006 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714392    4521 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 20:38:52.726006 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714394    4521 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 20:38:52.726006 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714397    4521 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 20:38:52.726006 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714400    4521 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 20:38:52.726006 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714403    4521 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 20:38:52.726006 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714406    4521 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 20:38:52.726006 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714411    4521 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 20:38:52.726006 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714414    4521 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 20:38:52.726006 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714416    4521 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 20:38:52.726006 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714419    4521 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 20:38:52.726006 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714421    4521 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 20:38:52.726006 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714424    4521 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 20:38:52.726006 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714427    4521 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 20:38:52.726006 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714429    4521 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 20:38:52.726006 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714432    4521 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 20:38:52.726006 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714435    4521 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 20:38:52.726707 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714437    4521 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 20:38:52.726707 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714441    4521 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 20:38:52.726707 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714445    4521 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 20:38:52.726707 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714448    4521 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 20:38:52.726707 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714451    4521 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 20:38:52.726707 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714453    4521 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 20:38:52.726707 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714456    4521 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 20:38:52.726707 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714459    4521 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 20:38:52.726707 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714461    4521 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 20:38:52.726707 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714464    4521 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 20:38:52.726707 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714467    4521 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 20:38:52.726707 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714469    4521 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 20:38:52.726707 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714472    4521 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 20:38:52.726707 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714474    4521 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 20:38:52.726707 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714477    4521 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 20:38:52.726707 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714479    4521 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 20:38:52.726707 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714482    4521 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 20:38:52.727133 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.714508    4521 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 20:38:52.727133 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714562    4521 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 20:38:52.727133 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714567    4521 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 20:38:52.727133 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714570    4521 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 20:38:52.727133 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714573    4521 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 20:38:52.727133 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714576    4521 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 20:38:52.727133 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714580    4521 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 20:38:52.727133 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714584    4521 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 20:38:52.727133 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714589    4521 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 20:38:52.727133 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714592    4521 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 20:38:52.727133 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714595    4521 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 20:38:52.727133 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714598    4521 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 20:38:52.727133 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714600    4521 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 20:38:52.727133 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714603    4521 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 20:38:52.727133 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714606    4521 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 20:38:52.727133 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714608    4521 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 20:38:52.727579 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714611    4521 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 20:38:52.727579 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714613    4521 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 20:38:52.727579 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714616    4521 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 20:38:52.727579 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714619    4521 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 20:38:52.727579 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714622    4521 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 20:38:52.727579 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714624    4521 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 20:38:52.727579 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714627    4521 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 20:38:52.727579 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714629    4521 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 20:38:52.727579 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714632    4521 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 20:38:52.727579 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714634    4521 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 20:38:52.727579 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714637    4521 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 20:38:52.727579 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714640    4521 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 20:38:52.727579 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714642    4521 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 20:38:52.727579 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714645    4521 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 20:38:52.727579 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714647    4521 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 20:38:52.727579 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714650    4521 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 20:38:52.727579 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714652    4521 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 20:38:52.727579 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714655    4521 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 20:38:52.727579 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714658    4521 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 20:38:52.727579 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714660    4521 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 20:38:52.728154 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714663    4521 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 20:38:52.728154 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714666    4521 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 20:38:52.728154 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714671    4521 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 20:38:52.728154 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714673    4521 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 20:38:52.728154 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714676    4521 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 20:38:52.728154 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714679    4521 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 20:38:52.728154 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714681    4521 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 20:38:52.728154 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714684    4521 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 20:38:52.728154 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714686    4521 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 20:38:52.728154 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714689    4521 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 20:38:52.728154 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714692    4521 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 20:38:52.728154 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714694    4521 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 20:38:52.728154 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714697    4521 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 20:38:52.728154 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714701    4521 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 20:38:52.728154 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714705    4521 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 20:38:52.728154 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714708    4521 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 20:38:52.728154 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714711    4521 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 20:38:52.728154 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714713    4521 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 20:38:52.728154 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714716    4521 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 20:38:52.728902 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714719    4521 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 20:38:52.728902 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714721    4521 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 20:38:52.728902 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714724    4521 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 20:38:52.728902 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714727    4521 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 20:38:52.728902 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714730    4521 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 20:38:52.728902 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714732    4521 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 20:38:52.728902 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714736    4521 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 20:38:52.728902 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714739    4521 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 20:38:52.728902 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714742    4521 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 20:38:52.728902 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714745    4521 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 20:38:52.728902 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714747    4521 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 20:38:52.728902 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714750    4521 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 20:38:52.728902 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714752    4521 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 20:38:52.728902 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714755    4521 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 20:38:52.728902 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714758    4521 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 20:38:52.728902 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714761    4521 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 20:38:52.728902 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714764    4521 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 20:38:52.728902 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714767    4521 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 20:38:52.728902 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714769    4521 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 20:38:52.728902 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714772    4521 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 20:38:52.729423 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714774    4521 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 20:38:52.729423 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714777    4521 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 20:38:52.729423 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714780    4521 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 20:38:52.729423 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714782    4521 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 20:38:52.729423 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714785    4521 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 20:38:52.729423 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714787    4521 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 20:38:52.729423 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714790    4521 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 20:38:52.729423 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714793    4521 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 20:38:52.729423 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714795    4521 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 20:38:52.729423 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714798    4521 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 20:38:52.729423 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714800    4521 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 20:38:52.729423 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714803    4521 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 20:38:52.729423 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714805    4521 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 20:38:52.729423 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714808    4521 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 20:38:52.729423 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714810    4521 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 20:38:52.729423 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714813    4521 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 20:38:52.729423 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714815    4521 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 20:38:52.729423 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714818    4521 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 20:38:52.729423 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714821    4521 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 20:38:52.729887 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714824    4521 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 20:38:52.729887 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714827    4521 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 20:38:52.729887 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714829    4521 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 20:38:52.729887 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714832    4521 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 20:38:52.729887 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714834    4521 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 20:38:52.729887 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714837    4521 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 20:38:52.729887 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714839    4521 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 20:38:52.729887 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714843    4521 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 20:38:52.729887 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714847    4521 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 20:38:52.729887 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714850    4521 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 20:38:52.729887 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714852    4521 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 20:38:52.729887 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714855    4521 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 20:38:52.729887 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.714858    4521 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 20:38:52.729887 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.718959    4521 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 20:38:52.729887 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.718974    4521 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 20:38:52.729887 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.718992    4521 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 20:38:52.729887 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.718996    4521 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 20:38:52.729887 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719052    4521 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 20:38:52.729887 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719058    4521 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 20:38:52.730368 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719061    4521 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 20:38:52.730368 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719065    4521 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 20:38:52.730368 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719068    4521 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 20:38:52.730368 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719070    4521 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 20:38:52.730368 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719074    4521 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 20:38:52.730368 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719076    4521 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 20:38:52.730368 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719079    4521 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 20:38:52.730368 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719084    4521 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 20:38:52.730368 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719088    4521 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 20:38:52.730368 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719092    4521 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 20:38:52.730368 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719095    4521 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 20:38:52.730368 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719098    4521 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 20:38:52.730368 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719102    4521 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 20:38:52.730368 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719104    4521 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 20:38:52.730368 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719107    4521 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 20:38:52.730368 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719109    4521 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 20:38:52.730368 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719113    4521 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 20:38:52.730368 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719116    4521 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 20:38:52.730368 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719118    4521 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 20:38:52.730836 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719121    4521 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 20:38:52.730836 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719124    4521 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 20:38:52.730836 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719128    4521 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 20:38:52.730836 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719130    4521 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 20:38:52.730836 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719133    4521 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 20:38:52.730836 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719136    4521 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 20:38:52.730836 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719138    4521 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 20:38:52.730836 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719142    4521 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 20:38:52.730836 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719144    4521 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 20:38:52.730836 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719147    4521 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 20:38:52.730836 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719150    4521 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 20:38:52.730836 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719152    4521 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 20:38:52.730836 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719155    4521 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 20:38:52.730836 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719158    4521 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 20:38:52.730836 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719160    4521 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 20:38:52.730836 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719163    4521 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 20:38:52.730836 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719165    4521 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 20:38:52.730836 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719168    4521 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 20:38:52.730836 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719170    4521 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 20:38:52.730836 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719173    4521 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 20:38:52.731452 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719176    4521 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 20:38:52.731452 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719178    4521 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 20:38:52.731452 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719181    4521 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 20:38:52.731452 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719183    4521 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 20:38:52.731452 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719186    4521 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 20:38:52.731452 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719189    4521 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 20:38:52.731452 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719191    4521 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 20:38:52.731452 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719194    4521 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 20:38:52.731452 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719197    4521 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 20:38:52.731452 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719200    4521 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 20:38:52.731452 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719203    4521 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 20:38:52.731452 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719206    4521 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 20:38:52.731452 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719209    4521 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 20:38:52.731452 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719211    4521 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 20:38:52.731452 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719214    4521 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 20:38:52.731452 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719217    4521 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 20:38:52.731452 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719220    4521 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 20:38:52.731452 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719223    4521 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 20:38:52.731452 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719225    4521 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 20:38:52.731964 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719228    4521 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 20:38:52.731964 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719230    4521 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 20:38:52.731964 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719233    4521 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 20:38:52.731964 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719236    4521 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 20:38:52.731964 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719238    4521 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 20:38:52.731964 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719241    4521 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 20:38:52.731964 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719243    4521 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 20:38:52.731964 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719246    4521 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 20:38:52.731964 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719248    4521 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 20:38:52.731964 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719252    4521 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 20:38:52.731964 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719255    4521 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 20:38:52.731964 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719258    4521 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 20:38:52.731964 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719261    4521 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 20:38:52.731964 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719263    4521 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 20:38:52.731964 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719266    4521 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 20:38:52.731964 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719269    4521 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 20:38:52.731964 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719271    4521 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 20:38:52.731964 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719274    4521 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 20:38:52.731964 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719276    4521 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 20:38:52.732439 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719279    4521 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 20:38:52.732439 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719282    4521 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 20:38:52.732439 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719285    4521 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 20:38:52.732439 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719287    4521 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 20:38:52.732439 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719289    4521 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 20:38:52.732439 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719292    4521 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 20:38:52.732439 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719295    4521 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 20:38:52.732439 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719298    4521 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 20:38:52.732439 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719300    4521 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 20:38:52.732439 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719303    4521 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 20:38:52.732439 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719306    4521 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 20:38:52.732439 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719309    4521 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 20:38:52.732439 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719311    4521 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 20:38:52.732439 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719314    4521 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 20:38:52.732439 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719316    4521 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 20:38:52.732439 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719319    4521 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 20:38:52.732439 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719338    4521 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 20:38:52.732439 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719341    4521 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 20:38:52.732439 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719344    4521 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 20:38:52.732439 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719347    4521 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 20:38:52.732911 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719350    4521 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 20:38:52.732911 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719352    4521 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 20:38:52.732911 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719355    4521 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 20:38:52.732911 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719358    4521 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 20:38:52.732911 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719360    4521 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 20:38:52.732911 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719362    4521 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 20:38:52.732911 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719365    4521 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 20:38:52.732911 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.719394    4521 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 20:38:52.732911 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719529    4521 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 20:38:52.732911 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719535    4521 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 20:38:52.732911 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719540    4521 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 20:38:52.732911 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719543    4521 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 20:38:52.732911 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719547    4521 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 20:38:52.732911 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719550    4521 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 20:38:52.732911 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719553    4521 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 20:38:52.733279 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719556    4521 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 20:38:52.733279 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719558    4521 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 20:38:52.733279 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719561    4521 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 20:38:52.733279 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719564    4521 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 20:38:52.733279 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719567    4521 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 20:38:52.733279 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719570    4521 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 20:38:52.733279 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719572    4521 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 20:38:52.733279 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719575    4521 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 20:38:52.733279 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719578    4521 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 20:38:52.733279 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719581    4521 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 20:38:52.733279 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719583    4521 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 20:38:52.733279 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719586    4521 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 20:38:52.733279 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719588    4521 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 20:38:52.733279 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719591    4521 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 20:38:52.733279 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719594    4521 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 20:38:52.733279 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719596    4521 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 20:38:52.733279 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719599    4521 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 20:38:52.733279 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719601    4521 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 20:38:52.733279 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719604    4521 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 20:38:52.733279 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719606    4521 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 20:38:52.733964 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719609    4521 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 20:38:52.733964 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719611    4521 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 20:38:52.733964 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719614    4521 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 20:38:52.733964 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719616    4521 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 20:38:52.733964 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719619    4521 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 20:38:52.733964 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719621    4521 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 20:38:52.733964 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719624    4521 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 20:38:52.733964 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719626    4521 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 20:38:52.733964 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719629    4521 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 20:38:52.733964 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719632    4521 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 20:38:52.733964 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719634    4521 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 20:38:52.733964 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719637    4521 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 20:38:52.733964 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719658    4521 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 20:38:52.733964 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719661    4521 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 20:38:52.733964 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719664    4521 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 20:38:52.733964 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719667    4521 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 20:38:52.733964 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719669    4521 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 20:38:52.733964 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719672    4521 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 20:38:52.733964 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719675    4521 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 20:38:52.733964 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719678    4521 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 20:38:52.734470 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719681    4521 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 20:38:52.734470 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719685    4521 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 20:38:52.734470 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719687    4521 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 20:38:52.734470 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719690    4521 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 20:38:52.734470 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719692    4521 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 20:38:52.734470 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719695    4521 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 20:38:52.734470 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719698    4521 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 20:38:52.734470 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719700    4521 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 20:38:52.734470 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719703    4521 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 20:38:52.734470 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719705    4521 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 20:38:52.734470 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719708    4521 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 20:38:52.734470 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719711    4521 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 20:38:52.734470 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719714    4521 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 20:38:52.734470 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719716    4521 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 20:38:52.734470 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719719    4521 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 20:38:52.734470 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719721    4521 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 20:38:52.734470 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719724    4521 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 20:38:52.734470 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719726    4521 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 20:38:52.734470 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719729    4521 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 20:38:52.734470 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719732    4521 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 20:38:52.734990 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719734    4521 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 20:38:52.734990 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719737    4521 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 20:38:52.734990 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719739    4521 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 20:38:52.734990 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719742    4521 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 20:38:52.734990 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719744    4521 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 20:38:52.734990 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719747    4521 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 20:38:52.734990 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719750    4521 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 20:38:52.734990 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719753    4521 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 20:38:52.734990 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719755    4521 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 20:38:52.734990 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719758    4521 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 20:38:52.734990 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719761    4521 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 20:38:52.734990 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719764    4521 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 20:38:52.734990 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719766    4521 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 20:38:52.734990 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719769    4521 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 20:38:52.734990 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719772    4521 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 20:38:52.734990 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719775    4521 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 20:38:52.734990 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719777    4521 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 20:38:52.734990 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719781    4521 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 20:38:52.734990 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719783    4521 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 20:38:52.734990 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719786    4521 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 20:38:52.735487 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719788    4521 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 20:38:52.735487 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719791    4521 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 20:38:52.735487 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719794    4521 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 20:38:52.735487 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719796    4521 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 20:38:52.735487 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719799    4521 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 20:38:52.735487 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719801    4521 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 20:38:52.735487 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719804    4521 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 20:38:52.735487 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719807    4521 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 20:38:52.735487 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719809    4521 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 20:38:52.735487 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719812    4521 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 20:38:52.735487 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719815    4521 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 20:38:52.735487 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719817    4521 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 20:38:52.735487 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719820    4521 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 20:38:52.735487 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719822    4521 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 20:38:52.735487 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719825    4521 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 20:38:52.735487 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719827    4521 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 20:38:52.735487 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719830    4521 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 20:38:52.735487 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719832    4521 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 20:38:52.735487 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719835    4521 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 20:38:52.735942 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.719860    4521 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 20:38:52.735942 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719901    4521 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 20:38:52.735942 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719906    4521 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 20:38:52.735942 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719909    4521 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 20:38:52.735942 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719912    4521 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 20:38:52.735942 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719915    4521 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 20:38:52.735942 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719917    4521 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 20:38:52.735942 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719920    4521 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 20:38:52.735942 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719923    4521 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 20:38:52.735942 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719926    4521 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 20:38:52.735942 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719929    4521 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 20:38:52.735942 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719932    4521 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 20:38:52.735942 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719935    4521 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 20:38:52.735942 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719938    4521 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 20:38:52.735942 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719941    4521 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 20:38:52.735942 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719943    4521 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 20:38:52.736386 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719946    4521 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 20:38:52.736386 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719948    4521 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 20:38:52.736386 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719951    4521 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 20:38:52.736386 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719953    4521 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 20:38:52.736386 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719956    4521 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 20:38:52.736386 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719958    4521 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 20:38:52.736386 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719961    4521 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 20:38:52.736386 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719963    4521 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 20:38:52.736386 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719966    4521 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 20:38:52.736386 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719968    4521 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 20:38:52.736386 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719971    4521 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 20:38:52.736386 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719973    4521 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 20:38:52.736386 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719975    4521 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 20:38:52.736386 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719978    4521 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 20:38:52.736386 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719980    4521 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 20:38:52.736386 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719983    4521 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 20:38:52.736386 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719986    4521 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 20:38:52.736386 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719989    4521 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 20:38:52.736386 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719991    4521 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 20:38:52.736386 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719994    4521 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 20:38:52.736867 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719996    4521 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 20:38:52.736867 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.719999    4521 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 20:38:52.736867 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.720001    4521 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 20:38:52.736867 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.720004    4521 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 20:38:52.736867 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.720006    4521 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 20:38:52.736867 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.720009    4521 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 20:38:52.736867 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.720014    4521 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 20:38:52.736867 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.720017    4521 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 20:38:52.736867 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.720020    4521 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 20:38:52.736867 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.720023    4521 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 20:38:52.736867 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.720025    4521 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 20:38:52.736867 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.720027    4521 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 20:38:52.736867 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.720030    4521 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 20:38:52.736867 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.720032    4521 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 20:38:52.736867 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.720035    4521 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 20:38:52.736867 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.720037    4521 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 20:38:52.736867 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.720040    4521 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 20:38:52.736867 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.720042    4521 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 20:38:52.736867 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.720045    4521 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 20:38:52.736867 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.720048    4521 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 20:38:52.737357 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.720050    4521 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 20:38:52.737357 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.720052    4521 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 20:38:52.737357 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.720055    4521 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 20:38:52.737357 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.720058    4521 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 20:38:52.737357 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.720061    4521 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 20:38:52.737357 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.720063    4521 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 20:38:52.737357 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.720066    4521 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 20:38:52.737357 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.720068    4521 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 20:38:52.737357 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.720071    4521 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 20:38:52.737357 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.720074    4521 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 20:38:52.737357 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.720077    4521 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 20:38:52.737357 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.720079    4521 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 20:38:52.737357 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.720082    4521 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 20:38:52.737357 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.720084    4521 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 20:38:52.737357 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.720087    4521 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 20:38:52.737357 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.720089    4521 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 20:38:52.737357 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.720092    4521 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 20:38:52.737357 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.720094    4521 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 20:38:52.737357 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.720098    4521 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 20:38:52.737811 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.720101    4521 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 20:38:52.737811 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.720104    4521 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 20:38:52.737811 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.720106    4521 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 20:38:52.737811 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.720109    4521 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 20:38:52.737811 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.720111    4521 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 20:38:52.737811 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.720113    4521 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 20:38:52.737811 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.720118    4521 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 20:38:52.737811 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.720122    4521 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 20:38:52.737811 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.720126    4521 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 20:38:52.737811 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.720129    4521 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 20:38:52.737811 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.720132    4521 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 20:38:52.737811 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.720134    4521 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 20:38:52.737811 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.720138    4521 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 20:38:52.737811 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.720140    4521 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 20:38:52.737811 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.720143    4521 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 20:38:52.737811 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.720145    4521 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 20:38:52.737811 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.720147    4521 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 20:38:52.737811 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.720150    4521 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 20:38:52.737811 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.720153    4521 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 20:38:52.738408 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.720155    4521 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 20:38:52.738408 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.720158    4521 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 20:38:52.738408 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.720162    4521 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 20:38:52.738408 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.720165    4521 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 20:38:52.738408 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.720168    4521 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 20:38:52.738408 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.720170    4521 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 20:38:52.738408 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.720173    4521 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 20:38:52.738408 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.720175    4521 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 20:38:52.738408 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.720178    4521 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 20:38:52.738408 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.720181    4521 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 20:38:52.738408 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.720183    4521 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 20:38:52.738408 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.720186    4521 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 20:38:52.738408 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:52.720192    4521 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 20:38:52.738408 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.720243    4521 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 20:38:52.738408 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.721236    4521 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Oct 30 20:38:52.738408 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.721295    4521 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Oct 30 20:38:52.738408 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.721481    4521 server.go:1014] "Starting client certificate rotation"
Oct 30 20:38:52.738408 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.721626    4521 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 20:38:52.738866 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.721674    4521 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 20:33:12 +0000 UTC" deadline="2028-08-16 22:21:29.983014308 +0000 UTC"
Oct 30 20:38:52.738866 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.721700    4521 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15745h42m37.261315834s"
Oct 30 20:38:52.738866 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.721962    4521 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 20:38:52.738866 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.723074    4521 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 20:38:52.738866 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.725989    4521 log.go:25] "Validated CRI v1 runtime API"
Oct 30 20:38:52.738866 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.727672    4521 log.go:25] "Validated CRI v1 image API"
Oct 30 20:38:52.738866 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.728483    4521 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 20:38:52.738866 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.730127    4521 fs.go:135] Filesystem UUIDs: map[548b60ec-451a-4113-b9ab-28b655e56cec:/dev/nvme0n1p3 62db3ad3-a467-4a0c-9144-c7af3bc828a7:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
Oct 30 20:38:52.739098 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.730155    4521 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/09d3ec791aaa85a086644433a7fab033324b526a8aec145332363a8c00682d5f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/09d3ec791aaa85a086644433a7fab033324b526a8aec145332363a8c00682d5f/userdata/shm major:0 minor:92 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0bcc74b6507dc0209f85ad2a19b8912b241efe513eb5ad35491b1737ca8a262c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0bcc74b6507dc0209f85ad2a19b8912b241efe513eb5ad35491b1737ca8a262c/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/228acd337e25401e4710ece12fd7e71c0d4581a209aef19e35790ee08ea50a0e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/228acd337e25401e4710ece12fd7e71c0d4581a209aef19e35790ee08ea50a0e/userdata/shm major:0 minor:77 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2d300712e711b03830289edf5cb38a7ecef41dad9f2269956fd7e6b3d7ada3b2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2d300712e711b03830289edf5cb38a7ecef41dad9f2269956fd7e6b3d7ada3b2/userdata/shm major:0 minor:88 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2f51971ec9317807796205a24a53a6fc151c1eb896de31293a0d96ab562b1818/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2f51971ec9317807796205a24a53a6fc151c1eb896de31293a0d96ab562b1818/userdata/shm major:0 minor:213 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/56461b767faa0395c16b73aa678e90856e30180670d31d772bd981dd7a51ceb5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/56461b767faa0395c16b73aa678e90856e30180670d31d772bd981dd7a51ceb5/userdata/shm major:0 minor:83 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/59229015d24ed2b2c8994862d50edd546044ceecc093c32fbf4815598cf2b7d2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/59229015d24ed2b2c8994862d50edd546044ceecc093c32fbf4815598cf2b7d2/userdata/shm major:0 minor:231 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7b9bba86d8bf288421f6b51c926a38211bfe44f6aca4598113aab535b8b73fb2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7b9bba86d8bf288421f6b51c926a38211bfe44f6aca4598113aab535b8b73fb2/userdata/shm major:0 minor:93 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/80a26258aed10d49cf54793b32a42a97bcfc09cf988ca045765d9d4d2d6d19ed/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/80a26258aed10d49cf54793b32a42a97bcfc09cf988ca045765d9d4d2d6d19ed/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8c7f18802822bdf57f6cb5a40c00900280dea799439b726ea8050dc19c8bce0d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8c7f18802822bdf57f6cb5a40c00900280dea799439b726ea8050dc19c8bce0d/userdata/shm major:0 minor:233 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/958c9bd7c6d62bcfd6effc523bd6f0835c3ba9f68387a4e1770cf0efdf7ae66b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/958c9bd7c6d62bcfd6effc523bd6f0835c3ba9f68387a4e1770cf0efdf7ae66b/userdata/shm major:0 minor:224 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/97605bd7a36f6cc05063a503084c058056dfa45b02a518afbf5402517fbc5e60/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/97605bd7a36f6cc05063a503084c058056dfa45b02a518afbf5402517fbc5e60/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a594a0250290aea1b704ebd3a04e2f28875c682787b876d44d14c101b9e9f03b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a594a0250290aea1b704ebd3a04e2f28875c682787b876d44d14c101b9e9f03b/userdata/shm major:0 minor:202 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c16277f3b4f3f62456078fbb6f9d926a1cdcf5849788f53a8bf3d1b1ff6d0028/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c16277f3b4f3f62456078fbb6f9d926a1cdcf5849788f53a8bf3d1b1ff6d0028/userdata/shm major:0 minor:49 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d8c56fe4c6222b749344a9775e4bb3134b1453511b9b5929dc9ea72f76dbe863/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d8c56fe4c6222b749344a9775e4bb3134b1453511b9b5929dc9ea72f76dbe863/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fa8ba53d948c6e44b70f9ccb9e5acd0fa6720e0200e37cabbeee987f9b243239/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fa8ba53d948c6e44b70f9ccb9e5acd0fa6720e0200e37cabbeee987f9b243239/userdata/shm major:0 minor:218 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1672e221-5127-44b8-bbc7-4d7b7ddc25c5/volumes/kubernetes.io~projected/kube-api-access-r8ggd:{mountpoint:/var/lib/kubelet/pods/1672e221-5127-44b8-bbc7-4d7b7ddc25c5/volumes/kubernetes.io~projected/kube-api-access-r8ggd major:0 minor:229 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1672e221-5127-44b8-bbc7-4d7b7ddc25c5/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/1672e221-5127-44b8-bbc7-4d7b7ddc25c5/volumes/kubernetes.io~secret/signing-key major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/30561fcc-3184-4116-b179-18941ff26a20/volumes/kubernetes.io~projected/kube-api-access-td58z:{mountpoint:/var/lib/kubelet/pods/30561fcc-3184-4116-b179-18941ff26a20/volumes/kubernetes.io~projected/kube-api-access-td58z major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/451ecbeb-c188-4e49-8e4e-08af1051ae0f/volumes/kubernetes.io~projected/kube-api-access-26z6g:{mountpoint:/var/lib/kubelet/pods/451ecbeb-c188-4e49-8e4e-08af1051ae0f/volumes/kubernetes.io~projected/kube-api-access-26z6g major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6fdde40e-3330-4b84-a470-6a7e6a536dce/volumes/kubernetes.io~projected/kube-api-access-p5hb6:{mountpoint:/var/lib/kubelet/pods/6fdde40e-3330-4b84-a470-6a7e6a536dce/volumes/kubernetes.io~projected/kube-api-access-p5hb6 major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/747c15be-3782-4065-a6dc-73d9fe7a4b4f/volumes/kubernetes.io~projected/kube-api-access-kxcww:{mountpoint:/var/lib/kubelet/pods/747c15be-3782-4065-a6dc-73d9fe7a4b4f/volumes/kubernetes.io~projected/kube-api-access-kxcww major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/747c15be-3782-4065-a6dc-73d9fe7a4b4f/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/747c15be-3782-4065-a6dc-73d9fe7a4b4f/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/76fb0a52-8bca-4610-9891-605df420cc7a/volumes/kubernetes.io~projected/kube-api-access-8flrx:{mountpoint:/var/lib/kubelet/pods/76fb0a52-8bca-4610-9891-605df420cc7a/volumes/kubernetes.io~projected/kube-api-access-8flrx major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7957ff19-e0fc-42a7-967d-6e96e4a88094/volumes/kubernetes.io~projected/kube-api-access-pqttz:{mountpoint:/var/lib/kubelet/pods/7957ff19-e0fc-42a7-967d-6e96e4a88094/volumes/kubernetes.io~projected/kube-api-access-pqttz major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/880d0493-e39d-4d9c-b58b-aa48c33aeaef/volumes/kubernetes.io~projected/kube-api-access-9fj44:{mountpoint:/var/lib/kubelet/pods/880d0493-e39d-4d9c-b58b-aa48c33aeaef/volumes/kubernetes.io~projected/kube-api-access-9fj44 major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9033c4e2-4586-4ec9-996b-21c412cfb8ba/volumes/kubernetes.io~projected/kube-api-access-2lvnm:{mountpoint:/var/lib/kubelet/pods/9033c4e2-4586-4ec9-996b-21c412cfb8ba/volumes/kubernetes.io~projected/kube-api-access-2lvnm major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/932cdb01-d506-4c20-9b4d-902da81e1f3c/volumes/kubernetes.io~projected/kube-api-access-dmvjx:{mountpoint:/var/lib/kubelet/pods/932cdb01-d506-4c20-9b4d-902da81e1f3c/volumes/kubernetes.io~projected/kube-api-access-dmvjx major:0 minor:230 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a03cdbf5-91a2-41eb-8c1e-3c0f40da2632/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/a03cdbf5-91a2-41eb-8c1e-3c0f40da2632/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c3967ccc-990a-42f5-845e-cbeec5e18be0/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/c3967ccc-990a-42f5-845e-cbeec5e18be0/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c3967ccc-990a-42f5-845e-cbeec5e18be0/volumes/kubernetes.io~projected/kube-api-access-z6nk5:{mountpoint:/var/lib/kubelet/pods/c3967ccc-990a-42f5-845e-cbeec5e18be0/volumes/kubernetes.io~projected/kube-api-access-z6nk5 major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c3967ccc-990a-42f5-845e-cbeec5e18be0/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/c3967ccc-990a-42f5-845e-cbeec5e18be0/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c6467dfc-7d9c-4ba8-ac2e-2ce19a74a0d4/volumes/kubernetes.io~projected/kube-api-access-4vxjc:{mountpoint:/var/lib/kubelet/pods/c6467dfc-7d9c-4ba8-ac2e-2ce19a74a0d4/volumes/kubernetes.io~projected/kube-api-access-4vxjc major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d585cbd8-cb89-4718-9af4-716f64cc2598/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/d585cbd8-cb89-4718-9af4-716f64cc2598/volumes/kubernetes.io~secret/agent-certs major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d5907106-812a-440a-b47d-f1889569d3b0/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/d5907106-812a-440a-b47d-f1889569d3b0/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d5907106-812a-440a-b47d-f1889569d3b0/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/d5907106-812a-440a-b47d-f1889569d3b0/volumes/kubernetes.io~empty-dir/tmp major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d5907106-812a-440a-b47d-f1889569d3b0/volumes/kubernetes.io~projected/kube-api-access-6h77j:{mountpoint:/var/lib/kubelet/pods/d5907106-812a-440a-b47d-f1889569d3b0/volumes/kubernetes.io~projected/kube-api-access-6h77j major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e584a74f-f82c-4a3f-81d6-0b93601980dd/volumes/kubernetes.io~projected/kube-api-access-smcsz:{mountpoint:/var/lib/kubelet/pods/e584a74f-f82c-4a3f-81d6-0b93601980dd/volumes/kubernetes.io~projected/kube-api-access-smcsz major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ee639a0c-5f44-4091-a876-fe9109934f74/volumes/kubernetes.io~projected/kube-api-access-gftnn:{mountpoint:/var/lib/kubelet/pods/ee639a0c-5f44-4091-a876-fe9109934f74/volumes/kubernetes.io~projected/kube-api-access-gftnn major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f831b1bf-4a1f-48b7-8242-85c063825277/volumes/kubernetes.io~projected/kube-api-access-mnt7j:{mountpoint:/var/lib/kubelet/pods/f831b1bf-4a1f-48b7-8242-85c063825277/volumes/kubernetes.io~projected/kube-api-access-mnt7j major:0 minor:63 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/5f5af1abf4297886477177ea534ff9f6631e7b911b8fbb41addffc9d14339191/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/83648cdbb9cf6eaa70f9d2d0f6d074a7706893faaa8dd8569abca4a7339640f5/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/8c13b06fdc09f07e3b49a122b67b1df865af6a46c087ad7e88b6e0677ce6a6e2/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/c166c584a4da166063e3eed4b92ee9a5fcf9e8b1a30255a17402cc05ea515405/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/68b46b49ef9035d3f2612c20bfcc79efd4f88e35da659ade783400af1ac83d6a/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/933fda8b6d9b6225354a42356fb91324aaa3ffcfdd8feed8fa6667399e429a37/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/a51da6c67c33a11a773779e8adf184718b3267659679e52354f964811ff55b72/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/d02c8fc8e228c3b2576bc504871955abcff57401a4235a92f7b3de19985a2b21/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/aa712752afdd07cc8c21eafa9a9280447ee6f6690001a5861194def1eeb853af/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/8b9637f051297e5b85ea989b1bf4ae1f1b4c6f32e133a36b24b2222169e87aa9/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-142:{mountpoint:/var/lib/containers/storage/overlay/1ce71d89de2a5691ab76b6cd1fe039dea699d17ae89eafaefe4b6fcc7ff0a570/merged major:0 minor:142 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/4d78f9cdb5198122bd40a290c77575c7adacc50ba911cacd2b8a31b333c3ad7e/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-149:{mountpoint:/var/lib/containers/storage/overlay/69619b0be793dd1b76a96241ddd6e750379ba683b402782001a2f49cd252c6d1/merged major:0 minor:149 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/f139826f3a768d6649ca9abd7d36b76e5c6aab5d668c14de020105663e0fea82/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/0a88bb3996e6548c5e0f12495a69fb0139439580d523bbe6584657313d116d3d/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/a19a7afe6326b37c2606a69777ee10a81e4c5d7f73594411c64d4f02ee57bc61/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/c7b7c08ca16d169b432157c94fbe44c121fe5bb573a751d4da8514610f94f7ee/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/c53fa07673e569a4c711f33549190294918767b6964471fe0a2757c625158f31/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-179:{mountpoint:/var/lib/containers/storage/overlay/66ddaac8498ed885596ad21b9c9b1e37a54ba338ae012aa9bf3e7b563f9ed5f5/merged major:0 minor:179 fsType:overlay blockSize:0} overlay_0-184:{mountpoint:/var/lib/containers/storage/overlay/a3ee7cb5880b93cf2e8e80a5e77157181d508d97abec4b4e76523704c07a9b11/merged major:0 minor:184 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/97731386a86fa31bd1969b7594f6857bfb674f2788d0cd56054c003bf2b4085f/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-204:{mountpoint:/var/lib/containers/storage/overlay/3754f0f94e3fb493841ff9cfb96ecc66b97d9c1afa7d8502f677b1fa5b5a35c2/merged major:0 minor:204 fsType:overlay blockSize:0} overlay_0-206:{mountpoint:/var/lib/containers/storage/overlay/f3735f1fc3e839d1aafcd435b8484e153465b040898302ef0bb40eacd38a15a6/merged major:0 minor:206 fsType:overlay blockSize:0} overlay_0-216:{mountpoint:/var/lib/containers/storage/overlay/60bbe5821a42f361a9bfe440adfd66bfa60e37c1ade0e12526ba557c722466e2/merged major:0 minor:216 fsType:overlay blockSize:0} overlay_0-220:{mountpoint:/var/lib/containers/storage/overlay/c95bb7efe1c24ad58eabafec0f61925c21f12f1e3496492cffd279d7baac0ad5/merged major:0 minor:220 fsType:overlay blockSize:0} overlay_0-227:{mountpoint:/var/lib/containers/storage/overlay/9ed80f5e383a1f0dd3cf02b980e9611a67b74111d23ec54891eb18462c5536b9/merged major:0 minor:227 fsType:overlay blockSize:0} overlay_0-235:{mountpoint:/var/lib/containers/storage/overlay/b1b81bbde3da8f4c7d8bbed95741322af2e0929c905405f02a960d50d7f7eadc/merged major:0 minor:235 fsType:overlay blockSize:0} overlay_0-237:{mountpoint:/var/lib/containers/storage/overlay/0a6a766224c65d7e52f60807f192bd3e20a1a9008163ee59d79a74191d565f9a/merged major:0 minor:237 fsType:overlay blockSize:0} overlay_0-239:{mountpoint:/var/lib/containers/storage/overlay/a4e26c99c34dd6b5a0ce36afe79b13d21398b6771ec69739093eb650ba1c5f86/merged major:0 minor:239 fsType:overlay blockSize:0} overlay_0-241:{mountpoint:/var/lib/containers/storage/overlay/78ad282ace595da3be8b2f6280200ebf1aace858a8b3517655485b9af024f8d2/merged major:0 minor:241 fsType:overlay blockSize:0} overlay_0-247:{mountpoint:/var/lib/containers/storage/overlay/1a45c0135382e20582c68ae46723ff70e0fe285456cfedf9c1b302839e357b6c/merged major:0 minor:247 fsType:overlay blockSize:0} overlay_0-253:{mountpoint:/var/lib/containers/storage/overlay/2fc779f2ae100d7c2c81d9761cb66779a938b644e003acb90af99f750b6c69eb/merged major:0 minor:253 fsType:overlay blockSize:0} overlay_0-50:{mountpoint:/var/lib/containers/storage/overlay/ec07e3a5d74af730de347cda0becc015e6bc83e7be19ca4059cf69e1da4989e7/merged major:0 minor:50 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/fd4762cec77d12917afd4c1727123461cff7971ed13d8dfeb1996b278c44f4b6/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/22ef507a2fc2a2b43d3d1c38e203d0b1a6d1728da38f6363cfc9dcd61b23ecb3/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/ead947c1410a079b7741511a8fc552a164e938dae98f439870b8be06434440c8/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-79:{mountpoint:/var/lib/containers/storage/overlay/5c88a315f0d1d3f2291e36700b52d5341569e316f21ffe04441ec1cc5a0a6601/merged major:0 minor:79 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/58295e9d4f16b9d7d6e62f1e1310c079761d02a980ec1b499ec3c83affd87a80/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/c4b96c4d10f02b5e63328e62ef64d751b29e843e493369355f56424fee2c337b/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/aeaf3456be932cb3ea37906ddb6d5e5caf8ad20f778fb326111d9d174072affc/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/e2ae18853db54393f0a3ea8cb5fd8e9e7c92af444673055e595078829f48efae/merged major:0 minor:96 fsType:overlay blockSize:0} overlay_0-98:{mountpoint:/var/lib/containers/storage/overlay/22874b550e559c5f08809faa9e43b847825b5d71093d9c6c82dba8dd8ee308e0/merged major:0 minor:98 fsType:overlay blockSize:0}]
Oct 30 20:38:52.739272 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.736814    4521 manager.go:217] Machine: {Timestamp:2026-10-30 20:38:52.736359264 +0000 UTC m=+0.081980493 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3217847 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2533b818d446c5b6c4b18c80a8918b SystemUUID:ec2533b8-18d4-46c5-b6c4-b18c80a8918b BootID:d8c270a2-8685-49ae-a7de-36fcea22edd0 Filesystems:[{Device:/var/lib/kubelet/pods/1672e221-5127-44b8-bbc7-4d7b7ddc25c5/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:226 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d8c56fe4c6222b749344a9775e4bb3134b1453511b9b5929dc9ea72f76dbe863/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c6467dfc-7d9c-4ba8-ac2e-2ce19a74a0d4/volumes/kubernetes.io~projected/kube-api-access-4vxjc DeviceMajor:0 DeviceMinor:59 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/880d0493-e39d-4d9c-b58b-aa48c33aeaef/volumes/kubernetes.io~projected/kube-api-access-9fj44 DeviceMajor:0 DeviceMinor:64 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/56461b767faa0395c16b73aa678e90856e30180670d31d772bd981dd7a51ceb5/userdata/shm DeviceMajor:0 DeviceMinor:83 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/747c15be-3782-4065-a6dc-73d9fe7a4b4f/volumes/kubernetes.io~projected/kube-api-access-kxcww DeviceMajor:0 DeviceMinor:197 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-50 DeviceMajor:0 DeviceMinor:50 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-204 DeviceMajor:0 DeviceMinor:204 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-216 DeviceMajor:0 DeviceMinor:216 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8c7f18802822bdf57f6cb5a40c00900280dea799439b726ea8050dc19c8bce0d/userdata/shm DeviceMajor:0 DeviceMinor:233 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/59229015d24ed2b2c8994862d50edd546044ceecc093c32fbf4815598cf2b7d2/userdata/shm DeviceMajor:0 DeviceMinor:231 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-237 DeviceMajor:0 DeviceMinor:237 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-239 DeviceMajor:0 DeviceMinor:239 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/80a26258aed10d49cf54793b32a42a97bcfc09cf988ca045765d9d4d2d6d19ed/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c3967ccc-990a-42f5-845e-cbeec5e18be0/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:55 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9033c4e2-4586-4ec9-996b-21c412cfb8ba/volumes/kubernetes.io~projected/kube-api-access-2lvnm DeviceMajor:0 DeviceMinor:222 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a594a0250290aea1b704ebd3a04e2f28875c682787b876d44d14c101b9e9f03b/userdata/shm DeviceMajor:0 DeviceMinor:202 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7957ff19-e0fc-42a7-967d-6e96e4a88094/volumes/kubernetes.io~projected/kube-api-access-pqttz DeviceMajor:0 DeviceMinor:215 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-235 DeviceMajor:0 DeviceMinor:235 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c16277f3b4f3f62456078fbb6f9d926a1cdcf5849788f53a8bf3d1b1ff6d0028/userdata/shm DeviceMajor:0 DeviceMinor:49 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-79 DeviceMajor:0 DeviceMinor:79 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-149 DeviceMajor:0 DeviceMinor:149 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a03cdbf5-91a2-41eb-8c1e-3c0f40da2632/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:208 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2f51971ec9317807796205a24a53a6fc151c1eb896de31293a0d96ab562b1818/userdata/shm DeviceMajor:0 DeviceMinor:213 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/451ecbeb-c188-4e49-8e4e-08af1051ae0f/volumes/kubernetes.io~projected/kube-api-access-26z6g DeviceMajor:0 DeviceMinor:223 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-247 DeviceMajor:0 DeviceMinor:247 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/228acd337e25401e4710ece12fd7e71c0d4581a209aef19e35790ee08ea50a0e/userdata/shm DeviceMajor:0 DeviceMinor:77 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2d300712e711b03830289edf5cb38a7ecef41dad9f2269956fd7e6b3d7ada3b2/userdata/shm DeviceMajor:0 DeviceMinor:88 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/932cdb01-d506-4c20-9b4d-902da81e1f3c/volumes/kubernetes.io~projected/kube-api-access-dmvjx DeviceMajor:0 DeviceMinor:230 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d5907106-812a-440a-b47d-f1889569d3b0/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:62 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fa8ba53d948c6e44b70f9ccb9e5acd0fa6720e0200e37cabbeee987f9b243239/userdata/shm DeviceMajor:0 DeviceMinor:218 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-253 DeviceMajor:0 DeviceMinor:253 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c3967ccc-990a-42f5-845e-cbeec5e18be0/volumes/kubernetes.io~projected/kube-api-access-z6nk5 DeviceMajor:0 DeviceMinor:60 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0bcc74b6507dc0209f85ad2a19b8912b241efe513eb5ad35491b1737ca8a262c/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-184 DeviceMajor:0 DeviceMinor:184 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d5907106-812a-440a-b47d-f1889569d3b0/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:61 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f831b1bf-4a1f-48b7-8242-85c063825277/volumes/kubernetes.io~projected/kube-api-access-mnt7j DeviceMajor:0 DeviceMinor:63 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/958c9bd7c6d62bcfd6effc523bd6f0835c3ba9f68387a4e1770cf0efdf7ae66b/userdata/shm DeviceMajor:0 DeviceMinor:224 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-241 DeviceMajor:0 DeviceMinor:241 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6fdde40e-3330-4b84-a470-6a7e6a536dce/volumes/kubernetes.io~projected/kube-api-access-p5hb6 DeviceMajor:0 DeviceMinor:57 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ee639a0c-5f44-4091-a876-fe9109934f74/volumes/kubernetes.io~projected/kube-api-access-gftnn DeviceMajor:0 DeviceMinor:58 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e584a74f-f82c-4a3f-81d6-0b93601980dd/volumes/kubernetes.io~projected/kube-api-access-smcsz DeviceMajor:0 DeviceMinor:67 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/747c15be-3782-4065-a6dc-73d9fe7a4b4f/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1672e221-5127-44b8-bbc7-4d7b7ddc25c5/volumes/kubernetes.io~projected/kube-api-access-r8ggd DeviceMajor:0 DeviceMinor:229 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-142 DeviceMajor:0 DeviceMinor:142 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-179 DeviceMajor:0 DeviceMinor:179 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/30561fcc-3184-4116-b179-18941ff26a20/volumes/kubernetes.io~projected/kube-api-access-td58z DeviceMajor:0 DeviceMinor:201 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-220 DeviceMajor:0 DeviceMinor:220 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-227 DeviceMajor:0 DeviceMinor:227 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/d585cbd8-cb89-4718-9af4-716f64cc2598/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:56 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d5907106-812a-440a-b47d-f1889569d3b0/volumes/kubernetes.io~projected/kube-api-access-6h77j DeviceMajor:0 DeviceMinor:65 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/76fb0a52-8bca-4610-9891-605df420cc7a/volumes/kubernetes.io~projected/kube-api-access-8flrx DeviceMajor:0 DeviceMinor:66 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/97605bd7a36f6cc05063a503084c058056dfa45b02a518afbf5402517fbc5e60/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/09d3ec791aaa85a086644433a7fab033324b526a8aec145332363a8c00682d5f/userdata/shm DeviceMajor:0 DeviceMinor:92 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-98 DeviceMajor:0 DeviceMinor:98 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7b9bba86d8bf288421f6b51c926a38211bfe44f6aca4598113aab535b8b73fb2/userdata/shm DeviceMajor:0 DeviceMinor:93 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c3967ccc-990a-42f5-845e-cbeec5e18be0/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-206 DeviceMajor:0 DeviceMinor:206 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:2f51971ec931780 MacAddress:d6:9b:45:9e:44:11 Speed:10000 Mtu:8901} {Name:59229015d24ed2b MacAddress:26:1c:05:90:52:c8 Speed:10000 Mtu:8901} {Name:8c7f18802822bdf MacAddress:2a:73:3a:03:b9:22 Speed:10000 Mtu:8901} {Name:958c9bd7c6d62bc MacAddress:fe:54:2c:ab:7b:24 Speed:10000 Mtu:8901} {Name:a594a0250290aea MacAddress:22:77:94:1d:e2:0f Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:f3:8d:bb:bf Speed:0 Mtu:9001} {Name:br-int MacAddress:d2:a1:22:1e:3f:7f Speed:0 Mtu:8901} {Name:ens5 MacAddress:02:ff:f3:8d:bb:bf Speed:0 Mtu:9001} {Name:fa8ba53d948c6e4 MacAddress:fa:68:26:7d:7c:97 Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:4a:d3:96:11:9e:dc Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:be:7a:29:98:9a:ba Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 20:38:52.739272 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.737070    4521 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 20:38:52.739272 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.737114    4521 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 20:38:52.739272 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.737437    4521 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 20:38:52.739272 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.737456    4521 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-125.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 20:38:52.739272 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.737603    4521 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 20:38:52.739272 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.737611    4521 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 20:38:52.739272 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.737622    4521 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 20:38:52.739272 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.737641    4521 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 20:38:52.739272 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.737734    4521 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 20:38:52.739272 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.737892    4521 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 20:38:52.739272 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.738009    4521 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 20:38:52.739272 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.738044    4521 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 20:38:52.739272 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.738053    4521 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 20:38:52.739272 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.738064    4521 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 20:38:52.739272 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.738073    4521 kubelet.go:405] "Adding apiserver pod source"
Oct 30 20:38:52.739272 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.738086    4521 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 20:38:52.739272 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.738762    4521 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 20:38:52.739272 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.738776    4521 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 20:38:52.739272 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.739264    4521 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 20:38:52.749829 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.749807    4521 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 20:38:52.749829 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.749824    4521 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 20:38:52.749980 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.749929    4521 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Oct 30 20:38:52.750169 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.750157    4521 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 20:38:52.750211 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.750187    4521 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 20:38:52.750288 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.750279    4521 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 20:38:52.750332 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.750293    4521 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 20:38:52.750332 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.750300    4521 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 20:38:52.750332 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.750306    4521 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 20:38:52.750332 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.750312    4521 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 20:38:52.750332 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.750318    4521 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 20:38:52.750488 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.750341    4521 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 20:38:52.750488 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.750347    4521 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 20:38:52.750488 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.750353    4521 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 20:38:52.750488 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.750360    4521 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 20:38:52.750488 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.750369    4521 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 20:38:52.750488 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.750379    4521 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 20:38:52.750488 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.750402    4521 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 20:38:52.750488 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.750408    4521 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 20:38:52.753974 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.753961    4521 server.go:1263] "Started kubelet"
Oct 30 20:38:52.756065 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.754055    4521 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 20:38:52.756065 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.754082    4521 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 20:38:52.756065 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.754127    4521 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 20:38:52.756065 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.754455    4521 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 20:38:52.756065 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.755309    4521 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 20:38:52.754535 ip-10-0-131-125 systemd[1]: Started Kubernetes Kubelet.
Oct 30 20:38:52.758420 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.756994    4521 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 20:38:52.758420 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.757011    4521 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 20:38:52.758420 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.757065    4521 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 20:33:14 +0000 UTC" deadline="2028-08-10 19:58:58.971312624 +0000 UTC"
Oct 30 20:38:52.758420 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.757096    4521 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15599h20m6.214219876s"
Oct 30 20:38:52.758420 ip-10-0-131-125 kubenswrapper[4521]: E1030 20:38:52.757139    4521 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-125.us-east-2.compute.internal\" not found"
Oct 30 20:38:52.758420 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.757172    4521 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 20:38:52.758420 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.757187    4521 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 20:38:52.758420 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.757218    4521 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 20:38:52.759289 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.759269    4521 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 20:38:52.759289 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.759262    4521 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="76fb0a52-8bca-4610-9891-605df420cc7a" volumeName="kubernetes.io/projected/76fb0a52-8bca-4610-9891-605df420cc7a-kube-api-access-8flrx" seLinuxMountContext=""
Oct 30 20:38:52.759440 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.759335    4521 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c3967ccc-990a-42f5-845e-cbeec5e18be0" volumeName="kubernetes.io/secret/c3967ccc-990a-42f5-845e-cbeec5e18be0-ovn-node-metrics-cert" seLinuxMountContext=""
Oct 30 20:38:52.759440 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.759352    4521 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d5907106-812a-440a-b47d-f1889569d3b0" volumeName="kubernetes.io/empty-dir/d5907106-812a-440a-b47d-f1889569d3b0-etc-tuned" seLinuxMountContext=""
Oct 30 20:38:52.759440 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.759361    4521 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e584a74f-f82c-4a3f-81d6-0b93601980dd" volumeName="kubernetes.io/configmap/e584a74f-f82c-4a3f-81d6-0b93601980dd-cni-binary-copy" seLinuxMountContext=""
Oct 30 20:38:52.759440 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.759369    4521 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e584a74f-f82c-4a3f-81d6-0b93601980dd" volumeName="kubernetes.io/configmap/e584a74f-f82c-4a3f-81d6-0b93601980dd-whereabouts-flatfile-configmap" seLinuxMountContext=""
Oct 30 20:38:52.759440 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.759377    4521 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1672e221-5127-44b8-bbc7-4d7b7ddc25c5" volumeName="kubernetes.io/projected/1672e221-5127-44b8-bbc7-4d7b7ddc25c5-kube-api-access-r8ggd" seLinuxMountContext=""
Oct 30 20:38:52.759440 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.759396    4521 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="451ecbeb-c188-4e49-8e4e-08af1051ae0f" volumeName="kubernetes.io/projected/451ecbeb-c188-4e49-8e4e-08af1051ae0f-kube-api-access-26z6g" seLinuxMountContext=""
Oct 30 20:38:52.759440 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.759417    4521 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="747c15be-3782-4065-a6dc-73d9fe7a4b4f" volumeName="kubernetes.io/secret/747c15be-3782-4065-a6dc-73d9fe7a4b4f-cert" seLinuxMountContext=""
Oct 30 20:38:52.759440 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.759433    4521 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7957ff19-e0fc-42a7-967d-6e96e4a88094" volumeName="kubernetes.io/projected/7957ff19-e0fc-42a7-967d-6e96e4a88094-kube-api-access-pqttz" seLinuxMountContext=""
Oct 30 20:38:52.759817 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.759445    4521 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="880d0493-e39d-4d9c-b58b-aa48c33aeaef" volumeName="kubernetes.io/configmap/880d0493-e39d-4d9c-b58b-aa48c33aeaef-serviceca" seLinuxMountContext=""
Oct 30 20:38:52.759817 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.759458    4521 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f831b1bf-4a1f-48b7-8242-85c063825277" volumeName="kubernetes.io/configmap/f831b1bf-4a1f-48b7-8242-85c063825277-iptables-alerter-script" seLinuxMountContext=""
Oct 30 20:38:52.759817 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.759470    4521 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="30561fcc-3184-4116-b179-18941ff26a20" volumeName="kubernetes.io/projected/30561fcc-3184-4116-b179-18941ff26a20-kube-api-access-td58z" seLinuxMountContext=""
Oct 30 20:38:52.759817 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.759481    4521 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a03cdbf5-91a2-41eb-8c1e-3c0f40da2632" volumeName="kubernetes.io/secret/a03cdbf5-91a2-41eb-8c1e-3c0f40da2632-original-pull-secret" seLinuxMountContext=""
Oct 30 20:38:52.759817 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.759494    4521 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e584a74f-f82c-4a3f-81d6-0b93601980dd" volumeName="kubernetes.io/configmap/e584a74f-f82c-4a3f-81d6-0b93601980dd-cni-sysctl-allowlist" seLinuxMountContext=""
Oct 30 20:38:52.759817 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.759511    4521 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ee639a0c-5f44-4091-a876-fe9109934f74" volumeName="kubernetes.io/configmap/ee639a0c-5f44-4091-a876-fe9109934f74-multus-daemon-config" seLinuxMountContext=""
Oct 30 20:38:52.759817 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.759523    4521 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f831b1bf-4a1f-48b7-8242-85c063825277" volumeName="kubernetes.io/projected/f831b1bf-4a1f-48b7-8242-85c063825277-kube-api-access-mnt7j" seLinuxMountContext=""
Oct 30 20:38:52.759817 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.759534    4521 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="30561fcc-3184-4116-b179-18941ff26a20" volumeName="kubernetes.io/configmap/30561fcc-3184-4116-b179-18941ff26a20-config-volume" seLinuxMountContext=""
Oct 30 20:38:52.759817 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.759549    4521 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="747c15be-3782-4065-a6dc-73d9fe7a4b4f" volumeName="kubernetes.io/projected/747c15be-3782-4065-a6dc-73d9fe7a4b4f-kube-api-access-kxcww" seLinuxMountContext=""
Oct 30 20:38:52.759817 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.759568    4521 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9033c4e2-4586-4ec9-996b-21c412cfb8ba" volumeName="kubernetes.io/configmap/9033c4e2-4586-4ec9-996b-21c412cfb8ba-trusted-ca-bundle" seLinuxMountContext=""
Oct 30 20:38:52.759817 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.759582    4521 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c6467dfc-7d9c-4ba8-ac2e-2ce19a74a0d4" volumeName="kubernetes.io/empty-dir/c6467dfc-7d9c-4ba8-ac2e-2ce19a74a0d4-tmp-dir" seLinuxMountContext=""
Oct 30 20:38:52.759817 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.759595    4521 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c6467dfc-7d9c-4ba8-ac2e-2ce19a74a0d4" volumeName="kubernetes.io/projected/c6467dfc-7d9c-4ba8-ac2e-2ce19a74a0d4-kube-api-access-4vxjc" seLinuxMountContext=""
Oct 30 20:38:52.759817 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.759607    4521 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1672e221-5127-44b8-bbc7-4d7b7ddc25c5" volumeName="kubernetes.io/configmap/1672e221-5127-44b8-bbc7-4d7b7ddc25c5-signing-cabundle" seLinuxMountContext=""
Oct 30 20:38:52.759817 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.759619    4521 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9033c4e2-4586-4ec9-996b-21c412cfb8ba" volumeName="kubernetes.io/configmap/9033c4e2-4586-4ec9-996b-21c412cfb8ba-service-ca-bundle" seLinuxMountContext=""
Oct 30 20:38:52.759817 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.759631    4521 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9033c4e2-4586-4ec9-996b-21c412cfb8ba" volumeName="kubernetes.io/empty-dir/9033c4e2-4586-4ec9-996b-21c412cfb8ba-archives-path" seLinuxMountContext=""
Oct 30 20:38:52.759817 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.759643    4521 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d5907106-812a-440a-b47d-f1889569d3b0" volumeName="kubernetes.io/empty-dir/d5907106-812a-440a-b47d-f1889569d3b0-tmp" seLinuxMountContext=""
Oct 30 20:38:52.759817 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.759658    4521 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e584a74f-f82c-4a3f-81d6-0b93601980dd" volumeName="kubernetes.io/projected/e584a74f-f82c-4a3f-81d6-0b93601980dd-kube-api-access-smcsz" seLinuxMountContext=""
Oct 30 20:38:52.759817 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.759685    4521 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6fdde40e-3330-4b84-a470-6a7e6a536dce" volumeName="kubernetes.io/projected/6fdde40e-3330-4b84-a470-6a7e6a536dce-kube-api-access-p5hb6" seLinuxMountContext=""
Oct 30 20:38:52.759817 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.759699    4521 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9033c4e2-4586-4ec9-996b-21c412cfb8ba" volumeName="kubernetes.io/projected/9033c4e2-4586-4ec9-996b-21c412cfb8ba-kube-api-access-2lvnm" seLinuxMountContext=""
Oct 30 20:38:52.759817 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.759716    4521 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="932cdb01-d506-4c20-9b4d-902da81e1f3c" volumeName="kubernetes.io/projected/932cdb01-d506-4c20-9b4d-902da81e1f3c-kube-api-access-dmvjx" seLinuxMountContext=""
Oct 30 20:38:52.759817 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.759729    4521 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c3967ccc-990a-42f5-845e-cbeec5e18be0" volumeName="kubernetes.io/configmap/c3967ccc-990a-42f5-845e-cbeec5e18be0-env-overrides" seLinuxMountContext=""
Oct 30 20:38:52.759817 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.759740    4521 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c3967ccc-990a-42f5-845e-cbeec5e18be0" volumeName="kubernetes.io/projected/c3967ccc-990a-42f5-845e-cbeec5e18be0-kube-api-access-z6nk5" seLinuxMountContext=""
Oct 30 20:38:52.759817 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.759755    4521 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d585cbd8-cb89-4718-9af4-716f64cc2598" volumeName="kubernetes.io/secret/d585cbd8-cb89-4718-9af4-716f64cc2598-agent-certs" seLinuxMountContext=""
Oct 30 20:38:52.759817 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.759768    4521 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ee639a0c-5f44-4091-a876-fe9109934f74" volumeName="kubernetes.io/projected/ee639a0c-5f44-4091-a876-fe9109934f74-kube-api-access-gftnn" seLinuxMountContext=""
Oct 30 20:38:52.759817 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.759780    4521 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1672e221-5127-44b8-bbc7-4d7b7ddc25c5" volumeName="kubernetes.io/secret/1672e221-5127-44b8-bbc7-4d7b7ddc25c5-signing-key" seLinuxMountContext=""
Oct 30 20:38:52.759817 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.759793    4521 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="30561fcc-3184-4116-b179-18941ff26a20" volumeName="kubernetes.io/empty-dir/30561fcc-3184-4116-b179-18941ff26a20-tmp-dir" seLinuxMountContext=""
Oct 30 20:38:52.759817 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.759805    4521 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c3967ccc-990a-42f5-845e-cbeec5e18be0" volumeName="kubernetes.io/configmap/c3967ccc-990a-42f5-845e-cbeec5e18be0-ovnkube-script-lib" seLinuxMountContext=""
Oct 30 20:38:52.759817 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.759816    4521 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d585cbd8-cb89-4718-9af4-716f64cc2598" volumeName="kubernetes.io/configmap/d585cbd8-cb89-4718-9af4-716f64cc2598-konnectivity-ca" seLinuxMountContext=""
Oct 30 20:38:52.759817 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.759829    4521 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d5907106-812a-440a-b47d-f1889569d3b0" volumeName="kubernetes.io/projected/d5907106-812a-440a-b47d-f1889569d3b0-kube-api-access-6h77j" seLinuxMountContext=""
Oct 30 20:38:52.761208 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.759841    4521 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="451ecbeb-c188-4e49-8e4e-08af1051ae0f" volumeName="kubernetes.io/empty-dir/451ecbeb-c188-4e49-8e4e-08af1051ae0f-data-volume" seLinuxMountContext=""
Oct 30 20:38:52.761208 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.759854    4521 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="880d0493-e39d-4d9c-b58b-aa48c33aeaef" volumeName="kubernetes.io/projected/880d0493-e39d-4d9c-b58b-aa48c33aeaef-kube-api-access-9fj44" seLinuxMountContext=""
Oct 30 20:38:52.761208 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.761152    4521 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c3967ccc-990a-42f5-845e-cbeec5e18be0" volumeName="kubernetes.io/configmap/c3967ccc-990a-42f5-845e-cbeec5e18be0-ovnkube-config" seLinuxMountContext=""
Oct 30 20:38:52.761208 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.761183    4521 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ee639a0c-5f44-4091-a876-fe9109934f74" volumeName="kubernetes.io/configmap/ee639a0c-5f44-4091-a876-fe9109934f74-cni-binary-copy" seLinuxMountContext=""
Oct 30 20:38:52.761208 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.761195    4521 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 20:38:52.761208 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.761204    4521 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 20:38:52.764725 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.762948    4521 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 20:38:52.764725 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.763190    4521 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 20:38:52.764725 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.763247    4521 factory.go:55] Registering systemd factory
Oct 30 20:38:52.764725 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.763274    4521 factory.go:223] Registration of the systemd container factory successfully
Oct 30 20:38:52.764725 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.763860    4521 factory.go:153] Registering CRI-O factory
Oct 30 20:38:52.764725 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.763874    4521 factory.go:223] Registration of the crio container factory successfully
Oct 30 20:38:52.764725 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.763932    4521 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 20:38:52.764725 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.763956    4521 factory.go:103] Registering Raw factory
Oct 30 20:38:52.764725 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.763970    4521 manager.go:1199] Started watching for new ooms in manager
Oct 30 20:38:52.764725 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.764628    4521 manager.go:319] Starting recovery of all containers
Oct 30 20:38:52.765389 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.765372    4521 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Oct 30 20:38:52.782335 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.782274    4521 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 20:38:52.785415 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.783748    4521 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 20:38:52.785415 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.783773    4521 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 20:38:52.785415 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.784494    4521 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 20:38:52.785415 ip-10-0-131-125 kubenswrapper[4521]: E1030 20:38:52.784561    4521 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 30 20:38:52.788791 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.788768    4521 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 20:38:52.795148 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.795120    4521 generic.go:356] "Generic (PLEG): container finished" podID="e584a74f-f82c-4a3f-81d6-0b93601980dd" containerID="61b1535f8b0316807697c01cdbb41fcedd10ad8d2a7ac47aa13770305df7ed07" exitCode=0
Oct 30 20:38:52.795148 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.795146    4521 generic.go:356] "Generic (PLEG): container finished" podID="e584a74f-f82c-4a3f-81d6-0b93601980dd" containerID="3aba57da98cc68bea670d9e7414c9400688d3c280c891ec42d6d23c535c9cccb" exitCode=0
Oct 30 20:38:52.795355 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.795155    4521 generic.go:356] "Generic (PLEG): container finished" podID="e584a74f-f82c-4a3f-81d6-0b93601980dd" containerID="276df7fca27c378acae9e76a6dae55982406f1c0ec0861c419827bbba91ce1af" exitCode=0
Oct 30 20:38:52.795355 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.795165    4521 generic.go:356] "Generic (PLEG): container finished" podID="e584a74f-f82c-4a3f-81d6-0b93601980dd" containerID="dd6a5a9517561387d40724145a33bba7fc7bbe9d81a0fa650dc9bfea18a1947d" exitCode=0
Oct 30 20:38:52.795355 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.795172    4521 generic.go:356] "Generic (PLEG): container finished" podID="e584a74f-f82c-4a3f-81d6-0b93601980dd" containerID="a5894271d75c9731ad7a8d91d9e8fb1fb8a131095dbdc896e06047c8b0ba638c" exitCode=0
Oct 30 20:38:52.795355 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.795180    4521 generic.go:356] "Generic (PLEG): container finished" podID="e584a74f-f82c-4a3f-81d6-0b93601980dd" containerID="ae29407fdc5eb0abdc0ebf1bb4f40785283580fd62971ffa93deef75ba422d49" exitCode=0
Oct 30 20:38:52.814252 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.814215    4521 generic.go:356] "Generic (PLEG): container finished" podID="73fe153e2da84792f2cfb4a59468aa21" containerID="674c2550b650706629cff4ef93d995f87a11905a35a5a10439e42553230ab3b1" exitCode=0
Oct 30 20:38:52.830914 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.830883    4521 manager.go:324] Recovery completed
Oct 30 20:38:52.847202 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.847133    4521 cpu_manager.go:226] "Starting" policy="none"
Oct 30 20:38:52.847202 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.847152    4521 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 20:38:52.847202 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.847173    4521 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 20:38:52.847408 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.847378    4521 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Oct 30 20:38:52.847499 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.847406    4521 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Oct 30 20:38:52.847499 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.847431    4521 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Oct 30 20:38:52.847499 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.847440    4521 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Oct 30 20:38:52.847499 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.847446    4521 policy_none.go:50] "Start"
Oct 30 20:38:52.847499 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.847455    4521 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 20:38:52.847499 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.847466    4521 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 20:38:52.848029 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.848015    4521 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Oct 30 20:38:52.848060 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.848030    4521 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Oct 30 20:38:52.848060 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.848037    4521 policy_none.go:44] "Start"
Oct 30 20:38:52.852001 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.851972    4521 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 20:38:52.854675 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.854662    4521 manager.go:341] "Starting Device Plugin manager"
Oct 30 20:38:52.854739 ip-10-0-131-125 kubenswrapper[4521]: E1030 20:38:52.854687    4521 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 20:38:52.854739 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.854695    4521 server.go:87] "Starting device plugin registration server"
Oct 30 20:38:52.854984 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.854970    4521 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 20:38:52.855055 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.854982    4521 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 20:38:52.855096 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.855079    4521 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 20:38:52.855229 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.855189    4521 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 20:38:52.855317 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.855242    4521 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 20:38:52.855317 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.855249    4521 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 20:38:52.855422 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.855307    4521 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T20:38:52.855221966Z","UUID":"c9ab9e73-dea7-4525-9b76-a80cc8ca6275","Handler":null,"Name":"","Endpoint":""}
Oct 30 20:38:52.856477 ip-10-0-131-125 kubenswrapper[4521]: E1030 20:38:52.856421    4521 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 20:38:52.856667 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.856653    4521 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 20:38:52.856733 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.856674    4521 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 20:38:52.885599 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.885557    4521 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-125.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-131-125.us-east-2.compute.internal"]
Oct 30 20:38:52.886029 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.885997    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-125.us-east-2.compute.internal" event={"ID":"cbbd659d200641ef129d88b1e4575eca","Type":"ContainerStarted","Data":"ed50815f5f70867e399607ccb259261aba2d787f157a0366878321ab09b25a4b"}
Oct 30 20:38:52.886117 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.886035    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-125.us-east-2.compute.internal" event={"ID":"cbbd659d200641ef129d88b1e4575eca","Type":"ContainerStarted","Data":"c16277f3b4f3f62456078fbb6f9d926a1cdcf5849788f53a8bf3d1b1ff6d0028"}
Oct 30 20:38:52.886117 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.886070    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-125.us-east-2.compute.internal" event={"ID":"73fe153e2da84792f2cfb4a59468aa21","Type":"ContainerStarted","Data":"ca65d62205a83e451c0fc33652d8bcf3c328603d7c9bd565aaad8247efdac4e7"}
Oct 30 20:38:52.886117 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.886082    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-125.us-east-2.compute.internal" event={"ID":"73fe153e2da84792f2cfb4a59468aa21","Type":"ContainerDied","Data":"674c2550b650706629cff4ef93d995f87a11905a35a5a10439e42553230ab3b1"}
Oct 30 20:38:52.886117 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.886093    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-125.us-east-2.compute.internal" event={"ID":"73fe153e2da84792f2cfb4a59468aa21","Type":"ContainerStarted","Data":"80a26258aed10d49cf54793b32a42a97bcfc09cf988ca045765d9d4d2d6d19ed"}
Oct 30 20:38:52.886117 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.886112    4521 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-125.us-east-2.compute.internal"
Oct 30 20:38:52.886385 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.886275    4521 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-125.us-east-2.compute.internal"
Oct 30 20:38:52.898625 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.898601    4521 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 20:38:52.898759 ip-10-0-131-125 kubenswrapper[4521]: E1030 20:38:52.898659    4521 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-131-125.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-125.us-east-2.compute.internal"
Oct 30 20:38:52.898994 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.898980    4521 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 20:38:52.899063 ip-10-0-131-125 kubenswrapper[4521]: E1030 20:38:52.899042    4521 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-131-125.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-125.us-east-2.compute.internal"
Oct 30 20:38:52.955102 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.955060    4521 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 20:38:52.957427 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.957405    4521 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-125.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 20:38:52.957546 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.957439    4521 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-125.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 20:38:52.957546 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.957450    4521 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-125.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 20:38:52.957546 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.957482    4521 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-131-125.us-east-2.compute.internal"
Oct 30 20:38:52.965609 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.965588    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/73fe153e2da84792f2cfb4a59468aa21-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-125.us-east-2.compute.internal\" (UID: \"73fe153e2da84792f2cfb4a59468aa21\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-125.us-east-2.compute.internal"
Oct 30 20:38:52.965689 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.965620    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/73fe153e2da84792f2cfb4a59468aa21-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-125.us-east-2.compute.internal\" (UID: \"73fe153e2da84792f2cfb4a59468aa21\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-125.us-east-2.compute.internal"
Oct 30 20:38:52.965689 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.965647    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/cbbd659d200641ef129d88b1e4575eca-config\") pod \"kube-apiserver-proxy-ip-10-0-131-125.us-east-2.compute.internal\" (UID: \"cbbd659d200641ef129d88b1e4575eca\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-125.us-east-2.compute.internal"
Oct 30 20:38:52.969941 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.969924    4521 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-131-125.us-east-2.compute.internal"
Oct 30 20:38:52.969993 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:52.969981    4521 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-131-125.us-east-2.compute.internal"
Oct 30 20:38:53.065810 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.065773    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/73fe153e2da84792f2cfb4a59468aa21-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-125.us-east-2.compute.internal\" (UID: \"73fe153e2da84792f2cfb4a59468aa21\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-125.us-east-2.compute.internal"
Oct 30 20:38:53.065810 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.065809    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/cbbd659d200641ef129d88b1e4575eca-config\") pod \"kube-apiserver-proxy-ip-10-0-131-125.us-east-2.compute.internal\" (UID: \"cbbd659d200641ef129d88b1e4575eca\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-125.us-east-2.compute.internal"
Oct 30 20:38:53.065986 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.065833    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/73fe153e2da84792f2cfb4a59468aa21-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-125.us-east-2.compute.internal\" (UID: \"73fe153e2da84792f2cfb4a59468aa21\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-125.us-east-2.compute.internal"
Oct 30 20:38:53.065986 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.065880    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/73fe153e2da84792f2cfb4a59468aa21-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-125.us-east-2.compute.internal\" (UID: \"73fe153e2da84792f2cfb4a59468aa21\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-125.us-east-2.compute.internal"
Oct 30 20:38:53.065986 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.065896    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/cbbd659d200641ef129d88b1e4575eca-config\") pod \"kube-apiserver-proxy-ip-10-0-131-125.us-east-2.compute.internal\" (UID: \"cbbd659d200641ef129d88b1e4575eca\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-125.us-east-2.compute.internal"
Oct 30 20:38:53.065986 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.065880    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/73fe153e2da84792f2cfb4a59468aa21-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-125.us-east-2.compute.internal\" (UID: \"73fe153e2da84792f2cfb4a59468aa21\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-125.us-east-2.compute.internal"
Oct 30 20:38:53.738419 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.738375    4521 apiserver.go:52] "Watching apiserver"
Oct 30 20:38:53.751571 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.751541    4521 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 20:38:53.751975 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.751956    4521 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-kxvqp","openshift-service-ca/service-ca-5fffc97bdf-f77w8","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-125.us-east-2.compute.internal","openshift-multus/multus-additional-cni-plugins-p5hfb","openshift-dns/node-resolver-4kpcm","openshift-insights/periodic-gathering-4f444-dttkw","openshift-multus/network-metrics-daemon-5bww9","openshift-network-operator/iptables-alerter-9p5zk","kube-system/kube-apiserver-proxy-ip-10-0-131-125.us-east-2.compute.internal","openshift-dns/dns-default-fblmm","kube-system/global-pull-secret-syncer-xkfsj","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kxsjg","openshift-cluster-node-tuning-operator/tuned-pllmn","kube-system/konnectivity-agent-qcrpl","openshift-image-registry/node-ca-2kdx9","openshift-ingress-canary/ingress-canary-q2wtm","openshift-kube-storage-version-migrator/migrator-66df5d496b-wqbfm","openshift-network-diagnostics/network-check-target-f4cch","openshift-insights/insights-runtime-extractor-lkn2k","openshift-ovn-kubernetes/ovnkube-node-mt2kl"]
Oct 30 20:38:53.752386 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.752371    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5bww9"
Oct 30 20:38:53.757461 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.757436    4521 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 20:38:53.757602 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.757582    4521 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 20:38:53.757712 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.757694    4521 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 20:38:53.757778 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.757709    4521 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-r2q8w\""
Oct 30 20:38:53.757822 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.757813    4521 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 20:38:53.758081 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.758062    4521 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 20:38:53.758423 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.758406    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-fblmm"
Oct 30 20:38:53.758577 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.758559    4521 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 20:38:53.758931 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.758912    4521 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-jrnvk\""
Oct 30 20:38:53.759019 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.759003    4521 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 20:38:53.759180 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.759164    4521 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 20:38:53.759230 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.759189    4521 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 20:38:53.759280 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.759255    4521 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 20:38:53.759375 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.759361    4521 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6zzqh\""
Oct 30 20:38:53.759479 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.759460    4521 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-nxdl7\""
Oct 30 20:38:53.759524 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.759477    4521 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-9r6zk\""
Oct 30 20:38:53.759613 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.759537    4521 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 20:38:53.759848 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.759819    4521 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 20:38:53.759978 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.759897    4521 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-mnk4k\""
Oct 30 20:38:53.760490 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.760470    4521 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 20:38:53.761415 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.761390    4521 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 20:38:53.761745 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.761720    4521 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-q55wr\""
Oct 30 20:38:53.761852 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.761772    4521 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 20:38:53.761852 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.761791    4521 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 20:38:53.761852 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.761807    4521 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 20:38:53.761852 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.761814    4521 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 20:38:53.761852 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.761847    4521 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 20:38:53.762076 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.761856    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-lkn2k"
Oct 30 20:38:53.762076 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.761886    4521 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 20:38:53.762076 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.761956    4521 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 20:38:53.762429 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.762410    4521 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 20:38:53.762501 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.762440    4521 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 20:38:53.762501 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.762439    4521 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 20:38:53.762501 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.762459    4521 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 20:38:53.762501 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.762479    4521 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 20:38:53.762639 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.762507    4521 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-gk5pt\""
Oct 30 20:38:53.762639 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.762522    4521 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 20:38:53.762930 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.762916    4521 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 20:38:53.762930 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.762921    4521 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 20:38:53.762930 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.762927    4521 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-r8zbf\""
Oct 30 20:38:53.763838 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.763819    4521 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-z6v2d\""
Oct 30 20:38:53.763838 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.763820    4521 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-cctqn\""
Oct 30 20:38:53.764152 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.764140    4521 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-wzc24\""
Oct 30 20:38:53.764402 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.764286    4521 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-9s58x\""
Oct 30 20:38:53.764402 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.764295    4521 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 20:38:53.764402 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.764302    4521 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 20:38:53.764402 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.764344    4521 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 20:38:53.764402 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.764358    4521 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 20:38:53.764402 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.764373    4521 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 20:38:53.764402 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.764384    4521 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 20:38:53.764402 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.764389    4521 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 20:38:53.764649 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.764419    4521 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-kxrnf\""
Oct 30 20:38:53.766207 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.766192    4521 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-8f8mm\""
Oct 30 20:38:53.766268 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.766212    4521 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-m94h2\""
Oct 30 20:38:53.767720 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.767707    4521 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Oct 30 20:38:53.767772 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.767722    4521 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Oct 30 20:38:53.767869 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.767854    4521 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Oct 30 20:38:53.767924 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.767891    4521 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 20:38:53.768019 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.768007    4521 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Oct 30 20:38:53.768019 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.768017    4521 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 20:38:53.768093 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.768030    4521 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Oct 30 20:38:53.768093 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.768080    4521 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 20:38:53.768466 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.768452    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c3967ccc-990a-42f5-845e-cbeec5e18be0-run-openvswitch\") pod \"ovnkube-node-mt2kl\" (UID: \"c3967ccc-990a-42f5-845e-cbeec5e18be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-mt2kl"
Oct 30 20:38:53.768499 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.768479    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d5907106-812a-440a-b47d-f1889569d3b0-etc-tuned\") pod \"tuned-pllmn\" (UID: \"d5907106-812a-440a-b47d-f1889569d3b0\") " pod="openshift-cluster-node-tuning-operator/tuned-pllmn"
Oct 30 20:38:53.768527 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.768498    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ee639a0c-5f44-4091-a876-fe9109934f74-multus-cni-dir\") pod \"multus-kxvqp\" (UID: \"ee639a0c-5f44-4091-a876-fe9109934f74\") " pod="openshift-multus/multus-kxvqp"
Oct 30 20:38:53.768527 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.768519    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e584a74f-f82c-4a3f-81d6-0b93601980dd-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-p5hfb\" (UID: \"e584a74f-f82c-4a3f-81d6-0b93601980dd\") " pod="openshift-multus/multus-additional-cni-plugins-p5hfb"
Oct 30 20:38:53.768612 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.768569    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vxjc\" (UniqueName: \"kubernetes.io/projected/c6467dfc-7d9c-4ba8-ac2e-2ce19a74a0d4-kube-api-access-4vxjc\") pod \"node-resolver-4kpcm\" (UID: \"c6467dfc-7d9c-4ba8-ac2e-2ce19a74a0d4\") " pod="openshift-dns/node-resolver-4kpcm"
Oct 30 20:38:53.768641 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.768631    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c3967ccc-990a-42f5-845e-cbeec5e18be0-host-cni-bin\") pod \"ovnkube-node-mt2kl\" (UID: \"c3967ccc-990a-42f5-845e-cbeec5e18be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-mt2kl"
Oct 30 20:38:53.768685 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.768671    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d5907106-812a-440a-b47d-f1889569d3b0-etc-tuned\") pod \"tuned-pllmn\" (UID: \"d5907106-812a-440a-b47d-f1889569d3b0\") " pod="openshift-cluster-node-tuning-operator/tuned-pllmn"
Oct 30 20:38:53.768724 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.768681    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c3967ccc-990a-42f5-845e-cbeec5e18be0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mt2kl\" (UID: \"c3967ccc-990a-42f5-845e-cbeec5e18be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-mt2kl"
Oct 30 20:38:53.768724 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.768715    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c3967ccc-990a-42f5-845e-cbeec5e18be0-ovnkube-script-lib\") pod \"ovnkube-node-mt2kl\" (UID: \"c3967ccc-990a-42f5-845e-cbeec5e18be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-mt2kl"
Oct 30 20:38:53.768875 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.768859    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ee639a0c-5f44-4091-a876-fe9109934f74-host-var-lib-cni-bin\") pod \"multus-kxvqp\" (UID: \"ee639a0c-5f44-4091-a876-fe9109934f74\") " pod="openshift-multus/multus-kxvqp"
Oct 30 20:38:53.768927 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.768879    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e584a74f-f82c-4a3f-81d6-0b93601980dd-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-p5hfb\" (UID: \"e584a74f-f82c-4a3f-81d6-0b93601980dd\") " pod="openshift-multus/multus-additional-cni-plugins-p5hfb"
Oct 30 20:38:53.768927 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.768917    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d5907106-812a-440a-b47d-f1889569d3b0-lib-modules\") pod \"tuned-pllmn\" (UID: \"d5907106-812a-440a-b47d-f1889569d3b0\") " pod="openshift-cluster-node-tuning-operator/tuned-pllmn"
Oct 30 20:38:53.769025 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.768929    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c3967ccc-990a-42f5-845e-cbeec5e18be0-ovnkube-script-lib\") pod \"ovnkube-node-mt2kl\" (UID: \"c3967ccc-990a-42f5-845e-cbeec5e18be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-mt2kl"
Oct 30 20:38:53.769025 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.768941    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c3967ccc-990a-42f5-845e-cbeec5e18be0-var-lib-openvswitch\") pod \"ovnkube-node-mt2kl\" (UID: \"c3967ccc-990a-42f5-845e-cbeec5e18be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-mt2kl"
Oct 30 20:38:53.769025 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.768973    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ee639a0c-5f44-4091-a876-fe9109934f74-cnibin\") pod \"multus-kxvqp\" (UID: \"ee639a0c-5f44-4091-a876-fe9109934f74\") " pod="openshift-multus/multus-kxvqp"
Oct 30 20:38:53.769025 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.769015    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ee639a0c-5f44-4091-a876-fe9109934f74-host-var-lib-kubelet\") pod \"multus-kxvqp\" (UID: \"ee639a0c-5f44-4091-a876-fe9109934f74\") " pod="openshift-multus/multus-kxvqp"
Oct 30 20:38:53.769158 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.769045    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d5907106-812a-440a-b47d-f1889569d3b0-tmp\") pod \"tuned-pllmn\" (UID: \"d5907106-812a-440a-b47d-f1889569d3b0\") " pod="openshift-cluster-node-tuning-operator/tuned-pllmn"
Oct 30 20:38:53.769158 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.769080    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/747c15be-3782-4065-a6dc-73d9fe7a4b4f-cert\") pod \"ingress-canary-q2wtm\" (UID: \"747c15be-3782-4065-a6dc-73d9fe7a4b4f\") " pod="openshift-ingress-canary/ingress-canary-q2wtm"
Oct 30 20:38:53.769158 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.769100    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/30561fcc-3184-4116-b179-18941ff26a20-metrics-tls\") pod \"dns-default-fblmm\" (UID: \"30561fcc-3184-4116-b179-18941ff26a20\") " pod="openshift-dns/dns-default-fblmm"
Oct 30 20:38:53.769158 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.769123    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c3967ccc-990a-42f5-845e-cbeec5e18be0-systemd-units\") pod \"ovnkube-node-mt2kl\" (UID: \"c3967ccc-990a-42f5-845e-cbeec5e18be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-mt2kl"
Oct 30 20:38:53.769158 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.769147    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d5907106-812a-440a-b47d-f1889569d3b0-tmp\") pod \"tuned-pllmn\" (UID: \"d5907106-812a-440a-b47d-f1889569d3b0\") " pod="openshift-cluster-node-tuning-operator/tuned-pllmn"
Oct 30 20:38:53.769349 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.769176    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c3967ccc-990a-42f5-845e-cbeec5e18be0-log-socket\") pod \"ovnkube-node-mt2kl\" (UID: \"c3967ccc-990a-42f5-845e-cbeec5e18be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-mt2kl"
Oct 30 20:38:53.769349 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.769218    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c3967ccc-990a-42f5-845e-cbeec5e18be0-host-run-ovn-kubernetes\") pod \"ovnkube-node-mt2kl\" (UID: \"c3967ccc-990a-42f5-845e-cbeec5e18be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-mt2kl"
Oct 30 20:38:53.769349 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.769255    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ee639a0c-5f44-4091-a876-fe9109934f74-host-run-k8s-cni-cncf-io\") pod \"multus-kxvqp\" (UID: \"ee639a0c-5f44-4091-a876-fe9109934f74\") " pod="openshift-multus/multus-kxvqp"
Oct 30 20:38:53.769349 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.769277    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d5907106-812a-440a-b47d-f1889569d3b0-etc-sysconfig\") pod \"tuned-pllmn\" (UID: \"d5907106-812a-440a-b47d-f1889569d3b0\") " pod="openshift-cluster-node-tuning-operator/tuned-pllmn"
Oct 30 20:38:53.769349 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.769292    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d5907106-812a-440a-b47d-f1889569d3b0-etc-sysctl-d\") pod \"tuned-pllmn\" (UID: \"d5907106-812a-440a-b47d-f1889569d3b0\") " pod="openshift-cluster-node-tuning-operator/tuned-pllmn"
Oct 30 20:38:53.769349 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.769310    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8flrx\" (UniqueName: \"kubernetes.io/projected/76fb0a52-8bca-4610-9891-605df420cc7a-kube-api-access-8flrx\") pod \"aws-ebs-csi-driver-node-kxsjg\" (UID: \"76fb0a52-8bca-4610-9891-605df420cc7a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kxsjg"
Oct 30 20:38:53.769626 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.769362    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6fdde40e-3330-4b84-a470-6a7e6a536dce-metrics-certs\") pod \"network-metrics-daemon-5bww9\" (UID: \"6fdde40e-3330-4b84-a470-6a7e6a536dce\") " pod="openshift-multus/network-metrics-daemon-5bww9"
Oct 30 20:38:53.769626 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.769410    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d5907106-812a-440a-b47d-f1889569d3b0-sys\") pod \"tuned-pllmn\" (UID: \"d5907106-812a-440a-b47d-f1889569d3b0\") " pod="openshift-cluster-node-tuning-operator/tuned-pllmn"
Oct 30 20:38:53.769626 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.769429    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/747c15be-3782-4065-a6dc-73d9fe7a4b4f-cert\") pod \"ingress-canary-q2wtm\" (UID: \"747c15be-3782-4065-a6dc-73d9fe7a4b4f\") " pod="openshift-ingress-canary/ingress-canary-q2wtm"
Oct 30 20:38:53.769626 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.769441    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a03cdbf5-91a2-41eb-8c1e-3c0f40da2632-dbus\") pod \"global-pull-secret-syncer-xkfsj\" (UID: \"a03cdbf5-91a2-41eb-8c1e-3c0f40da2632\") " pod="kube-system/global-pull-secret-syncer-xkfsj"
Oct 30 20:38:53.769626 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.769471    4521 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Oct 30 20:38:53.769626 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.769491    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/76fb0a52-8bca-4610-9891-605df420cc7a-socket-dir\") pod \"aws-ebs-csi-driver-node-kxsjg\" (UID: \"76fb0a52-8bca-4610-9891-605df420cc7a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kxsjg"
Oct 30 20:38:53.769626 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.769540    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/30561fcc-3184-4116-b179-18941ff26a20-config-volume\") pod \"dns-default-fblmm\" (UID: \"30561fcc-3184-4116-b179-18941ff26a20\") " pod="openshift-dns/dns-default-fblmm"
Oct 30 20:38:53.769626 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.769568    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5hb6\" (UniqueName: \"kubernetes.io/projected/6fdde40e-3330-4b84-a470-6a7e6a536dce-kube-api-access-p5hb6\") pod \"network-metrics-daemon-5bww9\" (UID: \"6fdde40e-3330-4b84-a470-6a7e6a536dce\") " pod="openshift-multus/network-metrics-daemon-5bww9"
Oct 30 20:38:53.769626 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.769617    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fj44\" (UniqueName: \"kubernetes.io/projected/880d0493-e39d-4d9c-b58b-aa48c33aeaef-kube-api-access-9fj44\") pod \"node-ca-2kdx9\" (UID: \"880d0493-e39d-4d9c-b58b-aa48c33aeaef\") " pod="openshift-image-registry/node-ca-2kdx9"
Oct 30 20:38:53.769933 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.769670    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnt7j\" (UniqueName: \"kubernetes.io/projected/f831b1bf-4a1f-48b7-8242-85c063825277-kube-api-access-mnt7j\") pod \"iptables-alerter-9p5zk\" (UID: \"f831b1bf-4a1f-48b7-8242-85c063825277\") " pod="openshift-network-operator/iptables-alerter-9p5zk"
Oct 30 20:38:53.769933 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.769791    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c3967ccc-990a-42f5-845e-cbeec5e18be0-env-overrides\") pod \"ovnkube-node-mt2kl\" (UID: \"c3967ccc-990a-42f5-845e-cbeec5e18be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-mt2kl"
Oct 30 20:38:53.769933 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.769840    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ee639a0c-5f44-4091-a876-fe9109934f74-cni-binary-copy\") pod \"multus-kxvqp\" (UID: \"ee639a0c-5f44-4091-a876-fe9109934f74\") " pod="openshift-multus/multus-kxvqp"
Oct 30 20:38:53.769933 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.769890    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ee639a0c-5f44-4091-a876-fe9109934f74-etc-kubernetes\") pod \"multus-kxvqp\" (UID: \"ee639a0c-5f44-4091-a876-fe9109934f74\") " pod="openshift-multus/multus-kxvqp"
Oct 30 20:38:53.769933 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.769889    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/30561fcc-3184-4116-b179-18941ff26a20-config-volume\") pod \"dns-default-fblmm\" (UID: \"30561fcc-3184-4116-b179-18941ff26a20\") " pod="openshift-dns/dns-default-fblmm"
Oct 30 20:38:53.769933 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.769931    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c3967ccc-990a-42f5-845e-cbeec5e18be0-env-overrides\") pod \"ovnkube-node-mt2kl\" (UID: \"c3967ccc-990a-42f5-845e-cbeec5e18be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-mt2kl"
Oct 30 20:38:53.770144 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.769945    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d5907106-812a-440a-b47d-f1889569d3b0-etc-modprobe-d\") pod \"tuned-pllmn\" (UID: \"d5907106-812a-440a-b47d-f1889569d3b0\") " pod="openshift-cluster-node-tuning-operator/tuned-pllmn"
Oct 30 20:38:53.770144 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.769971    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smcsz\" (UniqueName: \"kubernetes.io/projected/e584a74f-f82c-4a3f-81d6-0b93601980dd-kube-api-access-smcsz\") pod \"multus-additional-cni-plugins-p5hfb\" (UID: \"e584a74f-f82c-4a3f-81d6-0b93601980dd\") " pod="openshift-multus/multus-additional-cni-plugins-p5hfb"
Oct 30 20:38:53.770144 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.770012    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c6467dfc-7d9c-4ba8-ac2e-2ce19a74a0d4-hosts-file\") pod \"node-resolver-4kpcm\" (UID: \"c6467dfc-7d9c-4ba8-ac2e-2ce19a74a0d4\") " pod="openshift-dns/node-resolver-4kpcm"
Oct 30 20:38:53.770144 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.770052    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c3967ccc-990a-42f5-845e-cbeec5e18be0-host-slash\") pod \"ovnkube-node-mt2kl\" (UID: \"c3967ccc-990a-42f5-845e-cbeec5e18be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-mt2kl"
Oct 30 20:38:53.770144 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.770075    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ee639a0c-5f44-4091-a876-fe9109934f74-host-run-netns\") pod \"multus-kxvqp\" (UID: \"ee639a0c-5f44-4091-a876-fe9109934f74\") " pod="openshift-multus/multus-kxvqp"
Oct 30 20:38:53.770144 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.770123    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a03cdbf5-91a2-41eb-8c1e-3c0f40da2632-original-pull-secret\") pod \"global-pull-secret-syncer-xkfsj\" (UID: \"a03cdbf5-91a2-41eb-8c1e-3c0f40da2632\") " pod="kube-system/global-pull-secret-syncer-xkfsj"
Oct 30 20:38:53.770297 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.770148    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/76fb0a52-8bca-4610-9891-605df420cc7a-etc-selinux\") pod \"aws-ebs-csi-driver-node-kxsjg\" (UID: \"76fb0a52-8bca-4610-9891-605df420cc7a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kxsjg"
Oct 30 20:38:53.770297 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.770163    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ee639a0c-5f44-4091-a876-fe9109934f74-cni-binary-copy\") pod \"multus-kxvqp\" (UID: \"ee639a0c-5f44-4091-a876-fe9109934f74\") " pod="openshift-multus/multus-kxvqp"
Oct 30 20:38:53.770297 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.770180    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/76fb0a52-8bca-4610-9891-605df420cc7a-sys-fs\") pod \"aws-ebs-csi-driver-node-kxsjg\" (UID: \"76fb0a52-8bca-4610-9891-605df420cc7a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kxsjg"
Oct 30 20:38:53.770297 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.770246    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f831b1bf-4a1f-48b7-8242-85c063825277-host-slash\") pod \"iptables-alerter-9p5zk\" (UID: \"f831b1bf-4a1f-48b7-8242-85c063825277\") " pod="openshift-network-operator/iptables-alerter-9p5zk"
Oct 30 20:38:53.770297 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.770276    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c3967ccc-990a-42f5-845e-cbeec5e18be0-etc-openvswitch\") pod \"ovnkube-node-mt2kl\" (UID: \"c3967ccc-990a-42f5-845e-cbeec5e18be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-mt2kl"
Oct 30 20:38:53.770297 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.770292    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d5907106-812a-440a-b47d-f1889569d3b0-etc-sysctl-conf\") pod \"tuned-pllmn\" (UID: \"d5907106-812a-440a-b47d-f1889569d3b0\") " pod="openshift-cluster-node-tuning-operator/tuned-pllmn"
Oct 30 20:38:53.770487 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.770315    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-td58z\" (UniqueName: \"kubernetes.io/projected/30561fcc-3184-4116-b179-18941ff26a20-kube-api-access-td58z\") pod \"dns-default-fblmm\" (UID: \"30561fcc-3184-4116-b179-18941ff26a20\") " pod="openshift-dns/dns-default-fblmm"
Oct 30 20:38:53.770487 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.770358    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f831b1bf-4a1f-48b7-8242-85c063825277-iptables-alerter-script\") pod \"iptables-alerter-9p5zk\" (UID: \"f831b1bf-4a1f-48b7-8242-85c063825277\") " pod="openshift-network-operator/iptables-alerter-9p5zk"
Oct 30 20:38:53.770487 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.770376    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d5907106-812a-440a-b47d-f1889569d3b0-var-lib-kubelet\") pod \"tuned-pllmn\" (UID: \"d5907106-812a-440a-b47d-f1889569d3b0\") " pod="openshift-cluster-node-tuning-operator/tuned-pllmn"
Oct 30 20:38:53.770487 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.770392    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a03cdbf5-91a2-41eb-8c1e-3c0f40da2632-kubelet-config\") pod \"global-pull-secret-syncer-xkfsj\" (UID: \"a03cdbf5-91a2-41eb-8c1e-3c0f40da2632\") " pod="kube-system/global-pull-secret-syncer-xkfsj"
Oct 30 20:38:53.770487 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.770408    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxcww\" (UniqueName: \"kubernetes.io/projected/747c15be-3782-4065-a6dc-73d9fe7a4b4f-kube-api-access-kxcww\") pod \"ingress-canary-q2wtm\" (UID: \"747c15be-3782-4065-a6dc-73d9fe7a4b4f\") " pod="openshift-ingress-canary/ingress-canary-q2wtm"
Oct 30 20:38:53.770487 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.770453    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a03cdbf5-91a2-41eb-8c1e-3c0f40da2632-original-pull-secret\") pod \"global-pull-secret-syncer-xkfsj\" (UID: \"a03cdbf5-91a2-41eb-8c1e-3c0f40da2632\") " pod="kube-system/global-pull-secret-syncer-xkfsj"
Oct 30 20:38:53.770487 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.770464    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d585cbd8-cb89-4718-9af4-716f64cc2598-agent-certs\") pod \"konnectivity-agent-qcrpl\" (UID: \"d585cbd8-cb89-4718-9af4-716f64cc2598\") " pod="kube-system/konnectivity-agent-qcrpl"
Oct 30 20:38:53.770797 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.770489    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ee639a0c-5f44-4091-a876-fe9109934f74-multus-daemon-config\") pod \"multus-kxvqp\" (UID: \"ee639a0c-5f44-4091-a876-fe9109934f74\") " pod="openshift-multus/multus-kxvqp"
Oct 30 20:38:53.770797 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.770514    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ee639a0c-5f44-4091-a876-fe9109934f74-host-run-multus-certs\") pod \"multus-kxvqp\" (UID: \"ee639a0c-5f44-4091-a876-fe9109934f74\") " pod="openshift-multus/multus-kxvqp"
Oct 30 20:38:53.770797 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.770537    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d5907106-812a-440a-b47d-f1889569d3b0-run\") pod \"tuned-pllmn\" (UID: \"d5907106-812a-440a-b47d-f1889569d3b0\") " pod="openshift-cluster-node-tuning-operator/tuned-pllmn"
Oct 30 20:38:53.770797 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.770576    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e584a74f-f82c-4a3f-81d6-0b93601980dd-cnibin\") pod \"multus-additional-cni-plugins-p5hfb\" (UID: \"e584a74f-f82c-4a3f-81d6-0b93601980dd\") " pod="openshift-multus/multus-additional-cni-plugins-p5hfb"
Oct 30 20:38:53.770797 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.770599    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/880d0493-e39d-4d9c-b58b-aa48c33aeaef-serviceca\") pod \"node-ca-2kdx9\" (UID: \"880d0493-e39d-4d9c-b58b-aa48c33aeaef\") " pod="openshift-image-registry/node-ca-2kdx9"
Oct 30 20:38:53.770797 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.770608    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f831b1bf-4a1f-48b7-8242-85c063825277-iptables-alerter-script\") pod \"iptables-alerter-9p5zk\" (UID: \"f831b1bf-4a1f-48b7-8242-85c063825277\") " pod="openshift-network-operator/iptables-alerter-9p5zk"
Oct 30 20:38:53.770797 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.770620    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c3967ccc-990a-42f5-845e-cbeec5e18be0-node-log\") pod \"ovnkube-node-mt2kl\" (UID: \"c3967ccc-990a-42f5-845e-cbeec5e18be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-mt2kl"
Oct 30 20:38:53.770797 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.770651    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c3967ccc-990a-42f5-845e-cbeec5e18be0-ovnkube-config\") pod \"ovnkube-node-mt2kl\" (UID: \"c3967ccc-990a-42f5-845e-cbeec5e18be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-mt2kl"
Oct 30 20:38:53.770797 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.770764    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c3967ccc-990a-42f5-845e-cbeec5e18be0-ovn-node-metrics-cert\") pod \"ovnkube-node-mt2kl\" (UID: \"c3967ccc-990a-42f5-845e-cbeec5e18be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-mt2kl"
Oct 30 20:38:53.770797 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.770783    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ee639a0c-5f44-4091-a876-fe9109934f74-multus-daemon-config\") pod \"multus-kxvqp\" (UID: \"ee639a0c-5f44-4091-a876-fe9109934f74\") " pod="openshift-multus/multus-kxvqp"
Oct 30 20:38:53.770797 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.770771    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/880d0493-e39d-4d9c-b58b-aa48c33aeaef-serviceca\") pod \"node-ca-2kdx9\" (UID: \"880d0493-e39d-4d9c-b58b-aa48c33aeaef\") " pod="openshift-image-registry/node-ca-2kdx9"
Oct 30 20:38:53.770797 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.770799    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ee639a0c-5f44-4091-a876-fe9109934f74-multus-socket-dir-parent\") pod \"multus-kxvqp\" (UID: \"ee639a0c-5f44-4091-a876-fe9109934f74\") " pod="openshift-multus/multus-kxvqp"
Oct 30 20:38:53.771238 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.770819    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c3967ccc-990a-42f5-845e-cbeec5e18be0-ovnkube-config\") pod \"ovnkube-node-mt2kl\" (UID: \"c3967ccc-990a-42f5-845e-cbeec5e18be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-mt2kl"
Oct 30 20:38:53.771238 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.770829    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e584a74f-f82c-4a3f-81d6-0b93601980dd-cni-binary-copy\") pod \"multus-additional-cni-plugins-p5hfb\" (UID: \"e584a74f-f82c-4a3f-81d6-0b93601980dd\") " pod="openshift-multus/multus-additional-cni-plugins-p5hfb"
Oct 30 20:38:53.771238 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.770831    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d585cbd8-cb89-4718-9af4-716f64cc2598-agent-certs\") pod \"konnectivity-agent-qcrpl\" (UID: \"d585cbd8-cb89-4718-9af4-716f64cc2598\") " pod="kube-system/konnectivity-agent-qcrpl"
Oct 30 20:38:53.771238 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.770852    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e584a74f-f82c-4a3f-81d6-0b93601980dd-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-p5hfb\" (UID: \"e584a74f-f82c-4a3f-81d6-0b93601980dd\") " pod="openshift-multus/multus-additional-cni-plugins-p5hfb"
Oct 30 20:38:53.771238 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.770870    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6467dfc-7d9c-4ba8-ac2e-2ce19a74a0d4-tmp-dir\") pod \"node-resolver-4kpcm\" (UID: \"c6467dfc-7d9c-4ba8-ac2e-2ce19a74a0d4\") " pod="openshift-dns/node-resolver-4kpcm"
Oct 30 20:38:53.771238 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.770876    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c3967ccc-990a-42f5-845e-cbeec5e18be0-ovn-node-metrics-cert\") pod \"ovnkube-node-mt2kl\" (UID: \"c3967ccc-990a-42f5-845e-cbeec5e18be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-mt2kl"
Oct 30 20:38:53.771238 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.770884    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/76fb0a52-8bca-4610-9891-605df420cc7a-device-dir\") pod \"aws-ebs-csi-driver-node-kxsjg\" (UID: \"76fb0a52-8bca-4610-9891-605df420cc7a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kxsjg"
Oct 30 20:38:53.771238 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.770912    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c3967ccc-990a-42f5-845e-cbeec5e18be0-host-cni-netd\") pod \"ovnkube-node-mt2kl\" (UID: \"c3967ccc-990a-42f5-845e-cbeec5e18be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-mt2kl"
Oct 30 20:38:53.771238 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.770948    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6467dfc-7d9c-4ba8-ac2e-2ce19a74a0d4-tmp-dir\") pod \"node-resolver-4kpcm\" (UID: \"c6467dfc-7d9c-4ba8-ac2e-2ce19a74a0d4\") " pod="openshift-dns/node-resolver-4kpcm"
Oct 30 20:38:53.771238 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.770948    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6nk5\" (UniqueName: \"kubernetes.io/projected/c3967ccc-990a-42f5-845e-cbeec5e18be0-kube-api-access-z6nk5\") pod \"ovnkube-node-mt2kl\" (UID: \"c3967ccc-990a-42f5-845e-cbeec5e18be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-mt2kl"
Oct 30 20:38:53.771238 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.771037    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e584a74f-f82c-4a3f-81d6-0b93601980dd-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-p5hfb\" (UID: \"e584a74f-f82c-4a3f-81d6-0b93601980dd\") " pod="openshift-multus/multus-additional-cni-plugins-p5hfb"
Oct 30 20:38:53.771238 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.771104    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ee639a0c-5f44-4091-a876-fe9109934f74-system-cni-dir\") pod \"multus-kxvqp\" (UID: \"ee639a0c-5f44-4091-a876-fe9109934f74\") " pod="openshift-multus/multus-kxvqp"
Oct 30 20:38:53.771238 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.771185    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ee639a0c-5f44-4091-a876-fe9109934f74-host-var-lib-cni-multus\") pod \"multus-kxvqp\" (UID: \"ee639a0c-5f44-4091-a876-fe9109934f74\") " pod="openshift-multus/multus-kxvqp"
Oct 30 20:38:53.771238 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.771201    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e584a74f-f82c-4a3f-81d6-0b93601980dd-cni-binary-copy\") pod \"multus-additional-cni-plugins-p5hfb\" (UID: \"e584a74f-f82c-4a3f-81d6-0b93601980dd\") " pod="openshift-multus/multus-additional-cni-plugins-p5hfb"
Oct 30 20:38:53.771238 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.771209    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gftnn\" (UniqueName: \"kubernetes.io/projected/ee639a0c-5f44-4091-a876-fe9109934f74-kube-api-access-gftnn\") pod \"multus-kxvqp\" (UID: \"ee639a0c-5f44-4091-a876-fe9109934f74\") " pod="openshift-multus/multus-kxvqp"
Oct 30 20:38:53.771238 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.771225    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmvjx\" (UniqueName: \"kubernetes.io/projected/932cdb01-d506-4c20-9b4d-902da81e1f3c-kube-api-access-dmvjx\") pod \"migrator-66df5d496b-wqbfm\" (UID: \"932cdb01-d506-4c20-9b4d-902da81e1f3c\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-wqbfm"
Oct 30 20:38:53.772108 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.771247    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d5907106-812a-440a-b47d-f1889569d3b0-etc-kubernetes\") pod \"tuned-pllmn\" (UID: \"d5907106-812a-440a-b47d-f1889569d3b0\") " pod="openshift-cluster-node-tuning-operator/tuned-pllmn"
Oct 30 20:38:53.772108 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.771348    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e584a74f-f82c-4a3f-81d6-0b93601980dd-system-cni-dir\") pod \"multus-additional-cni-plugins-p5hfb\" (UID: \"e584a74f-f82c-4a3f-81d6-0b93601980dd\") " pod="openshift-multus/multus-additional-cni-plugins-p5hfb"
Oct 30 20:38:53.772108 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.771379    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c3967ccc-990a-42f5-845e-cbeec5e18be0-host-run-netns\") pod \"ovnkube-node-mt2kl\" (UID: \"c3967ccc-990a-42f5-845e-cbeec5e18be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-mt2kl"
Oct 30 20:38:53.772108 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.771401    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ee639a0c-5f44-4091-a876-fe9109934f74-multus-conf-dir\") pod \"multus-kxvqp\" (UID: \"ee639a0c-5f44-4091-a876-fe9109934f74\") " pod="openshift-multus/multus-kxvqp"
Oct 30 20:38:53.772108 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.771431    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d5907106-812a-440a-b47d-f1889569d3b0-etc-systemd\") pod \"tuned-pllmn\" (UID: \"d5907106-812a-440a-b47d-f1889569d3b0\") " pod="openshift-cluster-node-tuning-operator/tuned-pllmn"
Oct 30 20:38:53.772108 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.771476    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d5907106-812a-440a-b47d-f1889569d3b0-host\") pod \"tuned-pllmn\" (UID: \"d5907106-812a-440a-b47d-f1889569d3b0\") " pod="openshift-cluster-node-tuning-operator/tuned-pllmn"
Oct 30 20:38:53.772108 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.771535    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e584a74f-f82c-4a3f-81d6-0b93601980dd-tuning-conf-dir\") pod \"multus-additional-cni-plugins-p5hfb\" (UID: \"e584a74f-f82c-4a3f-81d6-0b93601980dd\") " pod="openshift-multus/multus-additional-cni-plugins-p5hfb"
Oct 30 20:38:53.772108 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.771558    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/76fb0a52-8bca-4610-9891-605df420cc7a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kxsjg\" (UID: \"76fb0a52-8bca-4610-9891-605df420cc7a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kxsjg"
Oct 30 20:38:53.772108 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.771581    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/76fb0a52-8bca-4610-9891-605df420cc7a-registration-dir\") pod \"aws-ebs-csi-driver-node-kxsjg\" (UID: \"76fb0a52-8bca-4610-9891-605df420cc7a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kxsjg"
Oct 30 20:38:53.772108 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.771598    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqttz\" (UniqueName: \"kubernetes.io/projected/7957ff19-e0fc-42a7-967d-6e96e4a88094-kube-api-access-pqttz\") pod \"network-check-target-f4cch\" (UID: \"7957ff19-e0fc-42a7-967d-6e96e4a88094\") " pod="openshift-network-diagnostics/network-check-target-f4cch"
Oct 30 20:38:53.772108 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.771715    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c3967ccc-990a-42f5-845e-cbeec5e18be0-host-kubelet\") pod \"ovnkube-node-mt2kl\" (UID: \"c3967ccc-990a-42f5-845e-cbeec5e18be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-mt2kl"
Oct 30 20:38:53.772108 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.771744    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c3967ccc-990a-42f5-845e-cbeec5e18be0-run-systemd\") pod \"ovnkube-node-mt2kl\" (UID: \"c3967ccc-990a-42f5-845e-cbeec5e18be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-mt2kl"
Oct 30 20:38:53.772108 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.771768    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6h77j\" (UniqueName: \"kubernetes.io/projected/d5907106-812a-440a-b47d-f1889569d3b0-kube-api-access-6h77j\") pod \"tuned-pllmn\" (UID: \"d5907106-812a-440a-b47d-f1889569d3b0\") " pod="openshift-cluster-node-tuning-operator/tuned-pllmn"
Oct 30 20:38:53.772108 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.771785    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30561fcc-3184-4116-b179-18941ff26a20-tmp-dir\") pod \"dns-default-fblmm\" (UID: \"30561fcc-3184-4116-b179-18941ff26a20\") " pod="openshift-dns/dns-default-fblmm"
Oct 30 20:38:53.772108 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.771863    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30561fcc-3184-4116-b179-18941ff26a20-tmp-dir\") pod \"dns-default-fblmm\" (UID: \"30561fcc-3184-4116-b179-18941ff26a20\") " pod="openshift-dns/dns-default-fblmm"
Oct 30 20:38:53.772108 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.771910    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d585cbd8-cb89-4718-9af4-716f64cc2598-konnectivity-ca\") pod \"konnectivity-agent-qcrpl\" (UID: \"d585cbd8-cb89-4718-9af4-716f64cc2598\") " pod="kube-system/konnectivity-agent-qcrpl"
Oct 30 20:38:53.772108 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.771972    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c3967ccc-990a-42f5-845e-cbeec5e18be0-run-ovn\") pod \"ovnkube-node-mt2kl\" (UID: \"c3967ccc-990a-42f5-845e-cbeec5e18be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-mt2kl"
Oct 30 20:38:53.772108 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.772041    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ee639a0c-5f44-4091-a876-fe9109934f74-hostroot\") pod \"multus-kxvqp\" (UID: \"ee639a0c-5f44-4091-a876-fe9109934f74\") " pod="openshift-multus/multus-kxvqp"
Oct 30 20:38:53.772108 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.772076    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/880d0493-e39d-4d9c-b58b-aa48c33aeaef-host\") pod \"node-ca-2kdx9\" (UID: \"880d0493-e39d-4d9c-b58b-aa48c33aeaef\") " pod="openshift-image-registry/node-ca-2kdx9"
Oct 30 20:38:53.772108 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.772112    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d585cbd8-cb89-4718-9af4-716f64cc2598-konnectivity-ca\") pod \"konnectivity-agent-qcrpl\" (UID: \"d585cbd8-cb89-4718-9af4-716f64cc2598\") " pod="kube-system/konnectivity-agent-qcrpl"
Oct 30 20:38:53.774258 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.774242    4521 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Oct 30 20:38:53.774684 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.774668    4521 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-6gb67\""
Oct 30 20:38:53.774741 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.774673    4521 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 20:38:53.774741 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.774701    4521 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-rcwsg\""
Oct 30 20:38:53.774741 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.774729    4521 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Oct 30 20:38:53.782967 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.782944    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8flrx\" (UniqueName: \"kubernetes.io/projected/76fb0a52-8bca-4610-9891-605df420cc7a-kube-api-access-8flrx\") pod \"aws-ebs-csi-driver-node-kxsjg\" (UID: \"76fb0a52-8bca-4610-9891-605df420cc7a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kxsjg"
Oct 30 20:38:53.783063 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.782947    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vxjc\" (UniqueName: \"kubernetes.io/projected/c6467dfc-7d9c-4ba8-ac2e-2ce19a74a0d4-kube-api-access-4vxjc\") pod \"node-resolver-4kpcm\" (UID: \"c6467dfc-7d9c-4ba8-ac2e-2ce19a74a0d4\") " pod="openshift-dns/node-resolver-4kpcm"
Oct 30 20:38:53.783647 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.783623    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fj44\" (UniqueName: \"kubernetes.io/projected/880d0493-e39d-4d9c-b58b-aa48c33aeaef-kube-api-access-9fj44\") pod \"node-ca-2kdx9\" (UID: \"880d0493-e39d-4d9c-b58b-aa48c33aeaef\") " pod="openshift-image-registry/node-ca-2kdx9"
Oct 30 20:38:53.783729 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.783682    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5hb6\" (UniqueName: \"kubernetes.io/projected/6fdde40e-3330-4b84-a470-6a7e6a536dce-kube-api-access-p5hb6\") pod \"network-metrics-daemon-5bww9\" (UID: \"6fdde40e-3330-4b84-a470-6a7e6a536dce\") " pod="openshift-multus/network-metrics-daemon-5bww9"
Oct 30 20:38:53.783729 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.783703    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnt7j\" (UniqueName: \"kubernetes.io/projected/f831b1bf-4a1f-48b7-8242-85c063825277-kube-api-access-mnt7j\") pod \"iptables-alerter-9p5zk\" (UID: \"f831b1bf-4a1f-48b7-8242-85c063825277\") " pod="openshift-network-operator/iptables-alerter-9p5zk"
Oct 30 20:38:53.783729 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.783720    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-td58z\" (UniqueName: \"kubernetes.io/projected/30561fcc-3184-4116-b179-18941ff26a20-kube-api-access-td58z\") pod \"dns-default-fblmm\" (UID: \"30561fcc-3184-4116-b179-18941ff26a20\") " pod="openshift-dns/dns-default-fblmm"
Oct 30 20:38:53.784607 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.784589    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gftnn\" (UniqueName: \"kubernetes.io/projected/ee639a0c-5f44-4091-a876-fe9109934f74-kube-api-access-gftnn\") pod \"multus-kxvqp\" (UID: \"ee639a0c-5f44-4091-a876-fe9109934f74\") " pod="openshift-multus/multus-kxvqp"
Oct 30 20:38:53.784748 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.784732    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6nk5\" (UniqueName: \"kubernetes.io/projected/c3967ccc-990a-42f5-845e-cbeec5e18be0-kube-api-access-z6nk5\") pod \"ovnkube-node-mt2kl\" (UID: \"c3967ccc-990a-42f5-845e-cbeec5e18be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-mt2kl"
Oct 30 20:38:53.787335 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.787308    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6h77j\" (UniqueName: \"kubernetes.io/projected/d5907106-812a-440a-b47d-f1889569d3b0-kube-api-access-6h77j\") pod \"tuned-pllmn\" (UID: \"d5907106-812a-440a-b47d-f1889569d3b0\") " pod="openshift-cluster-node-tuning-operator/tuned-pllmn"
Oct 30 20:38:53.787673 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.787654    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqttz\" (UniqueName: \"kubernetes.io/projected/7957ff19-e0fc-42a7-967d-6e96e4a88094-kube-api-access-pqttz\") pod \"network-check-target-f4cch\" (UID: \"7957ff19-e0fc-42a7-967d-6e96e4a88094\") " pod="openshift-network-diagnostics/network-check-target-f4cch"
Oct 30 20:38:53.787793 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.787777    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxcww\" (UniqueName: \"kubernetes.io/projected/747c15be-3782-4065-a6dc-73d9fe7a4b4f-kube-api-access-kxcww\") pod \"ingress-canary-q2wtm\" (UID: \"747c15be-3782-4065-a6dc-73d9fe7a4b4f\") " pod="openshift-ingress-canary/ingress-canary-q2wtm"
Oct 30 20:38:53.787978 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.787959    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmvjx\" (UniqueName: \"kubernetes.io/projected/932cdb01-d506-4c20-9b4d-902da81e1f3c-kube-api-access-dmvjx\") pod \"migrator-66df5d496b-wqbfm\" (UID: \"932cdb01-d506-4c20-9b4d-902da81e1f3c\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-wqbfm"
Oct 30 20:38:53.792881 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.792861    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smcsz\" (UniqueName: \"kubernetes.io/projected/e584a74f-f82c-4a3f-81d6-0b93601980dd-kube-api-access-smcsz\") pod \"multus-additional-cni-plugins-p5hfb\" (UID: \"e584a74f-f82c-4a3f-81d6-0b93601980dd\") " pod="openshift-multus/multus-additional-cni-plugins-p5hfb"
Oct 30 20:38:53.823388 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.823360    4521 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-125.us-east-2.compute.internal"
Oct 30 20:38:53.837291 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.837262    4521 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 20:38:53.837409 ip-10-0-131-125 kubenswrapper[4521]: E1030 20:38:53.837317    4521 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-131-125.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-125.us-east-2.compute.internal"
Oct 30 20:38:53.858464 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.858443    4521 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 20:38:53.872428 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.872398    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/451ecbeb-c188-4e49-8e4e-08af1051ae0f-data-volume\") pod \"insights-runtime-extractor-lkn2k\" (UID: \"451ecbeb-c188-4e49-8e4e-08af1051ae0f\") " pod="openshift-insights/insights-runtime-extractor-lkn2k"
Oct 30 20:38:53.872428 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.872430    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26z6g\" (UniqueName: \"kubernetes.io/projected/451ecbeb-c188-4e49-8e4e-08af1051ae0f-kube-api-access-26z6g\") pod \"insights-runtime-extractor-lkn2k\" (UID: \"451ecbeb-c188-4e49-8e4e-08af1051ae0f\") " pod="openshift-insights/insights-runtime-extractor-lkn2k"
Oct 30 20:38:53.872590 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.872446    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9033c4e2-4586-4ec9-996b-21c412cfb8ba-trusted-ca-bundle\") pod \"periodic-gathering-4f444-dttkw\" (UID: \"9033c4e2-4586-4ec9-996b-21c412cfb8ba\") " pod="openshift-insights/periodic-gathering-4f444-dttkw"
Oct 30 20:38:53.872590 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.872462    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c3967ccc-990a-42f5-845e-cbeec5e18be0-run-ovn\") pod \"ovnkube-node-mt2kl\" (UID: \"c3967ccc-990a-42f5-845e-cbeec5e18be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-mt2kl"
Oct 30 20:38:53.872590 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.872477    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ee639a0c-5f44-4091-a876-fe9109934f74-hostroot\") pod \"multus-kxvqp\" (UID: \"ee639a0c-5f44-4091-a876-fe9109934f74\") " pod="openshift-multus/multus-kxvqp"
Oct 30 20:38:53.872590 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.872492    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/880d0493-e39d-4d9c-b58b-aa48c33aeaef-host\") pod \"node-ca-2kdx9\" (UID: \"880d0493-e39d-4d9c-b58b-aa48c33aeaef\") " pod="openshift-image-registry/node-ca-2kdx9"
Oct 30 20:38:53.872590 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.872506    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8ggd\" (UniqueName: \"kubernetes.io/projected/1672e221-5127-44b8-bbc7-4d7b7ddc25c5-kube-api-access-r8ggd\") pod \"service-ca-5fffc97bdf-f77w8\" (UID: \"1672e221-5127-44b8-bbc7-4d7b7ddc25c5\") " pod="openshift-service-ca/service-ca-5fffc97bdf-f77w8"
Oct 30 20:38:53.872590 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.872520    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c3967ccc-990a-42f5-845e-cbeec5e18be0-run-openvswitch\") pod \"ovnkube-node-mt2kl\" (UID: \"c3967ccc-990a-42f5-845e-cbeec5e18be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-mt2kl"
Oct 30 20:38:53.872590 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.872557    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/451ecbeb-c188-4e49-8e4e-08af1051ae0f-data-volume\") pod \"insights-runtime-extractor-lkn2k\" (UID: \"451ecbeb-c188-4e49-8e4e-08af1051ae0f\") " pod="openshift-insights/insights-runtime-extractor-lkn2k"
Oct 30 20:38:53.872590 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.872572    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c3967ccc-990a-42f5-845e-cbeec5e18be0-run-ovn\") pod \"ovnkube-node-mt2kl\" (UID: \"c3967ccc-990a-42f5-845e-cbeec5e18be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-mt2kl"
Oct 30 20:38:53.872590 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.872542    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ee639a0c-5f44-4091-a876-fe9109934f74-multus-cni-dir\") pod \"multus-kxvqp\" (UID: \"ee639a0c-5f44-4091-a876-fe9109934f74\") " pod="openshift-multus/multus-kxvqp"
Oct 30 20:38:53.872590 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.872575    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/880d0493-e39d-4d9c-b58b-aa48c33aeaef-host\") pod \"node-ca-2kdx9\" (UID: \"880d0493-e39d-4d9c-b58b-aa48c33aeaef\") " pod="openshift-image-registry/node-ca-2kdx9"
Oct 30 20:38:53.872969 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.872581    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ee639a0c-5f44-4091-a876-fe9109934f74-hostroot\") pod \"multus-kxvqp\" (UID: \"ee639a0c-5f44-4091-a876-fe9109934f74\") " pod="openshift-multus/multus-kxvqp"
Oct 30 20:38:53.872969 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.872616    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c3967ccc-990a-42f5-845e-cbeec5e18be0-run-openvswitch\") pod \"ovnkube-node-mt2kl\" (UID: \"c3967ccc-990a-42f5-845e-cbeec5e18be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-mt2kl"
Oct 30 20:38:53.872969 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.872620    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ee639a0c-5f44-4091-a876-fe9109934f74-multus-cni-dir\") pod \"multus-kxvqp\" (UID: \"ee639a0c-5f44-4091-a876-fe9109934f74\") " pod="openshift-multus/multus-kxvqp"
Oct 30 20:38:53.872969 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.872629    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c3967ccc-990a-42f5-845e-cbeec5e18be0-host-cni-bin\") pod \"ovnkube-node-mt2kl\" (UID: \"c3967ccc-990a-42f5-845e-cbeec5e18be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-mt2kl"
Oct 30 20:38:53.873135 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.873101    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c3967ccc-990a-42f5-845e-cbeec5e18be0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mt2kl\" (UID: \"c3967ccc-990a-42f5-845e-cbeec5e18be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-mt2kl"
Oct 30 20:38:53.873227 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.873207    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c3967ccc-990a-42f5-845e-cbeec5e18be0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mt2kl\" (UID: \"c3967ccc-990a-42f5-845e-cbeec5e18be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-mt2kl"
Oct 30 20:38:53.873277 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.873212    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9033c4e2-4586-4ec9-996b-21c412cfb8ba-trusted-ca-bundle\") pod \"periodic-gathering-4f444-dttkw\" (UID: \"9033c4e2-4586-4ec9-996b-21c412cfb8ba\") " pod="openshift-insights/periodic-gathering-4f444-dttkw"
Oct 30 20:38:53.873353 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.873290    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ee639a0c-5f44-4091-a876-fe9109934f74-host-var-lib-cni-bin\") pod \"multus-kxvqp\" (UID: \"ee639a0c-5f44-4091-a876-fe9109934f74\") " pod="openshift-multus/multus-kxvqp"
Oct 30 20:38:53.873405 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.873373    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d5907106-812a-440a-b47d-f1889569d3b0-lib-modules\") pod \"tuned-pllmn\" (UID: \"d5907106-812a-440a-b47d-f1889569d3b0\") " pod="openshift-cluster-node-tuning-operator/tuned-pllmn"
Oct 30 20:38:53.873405 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.873391    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c3967ccc-990a-42f5-845e-cbeec5e18be0-host-cni-bin\") pod \"ovnkube-node-mt2kl\" (UID: \"c3967ccc-990a-42f5-845e-cbeec5e18be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-mt2kl"
Oct 30 20:38:53.873509 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.873434    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/451ecbeb-c188-4e49-8e4e-08af1051ae0f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lkn2k\" (UID: \"451ecbeb-c188-4e49-8e4e-08af1051ae0f\") " pod="openshift-insights/insights-runtime-extractor-lkn2k"
Oct 30 20:38:53.873509 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.873458    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ee639a0c-5f44-4091-a876-fe9109934f74-host-var-lib-cni-bin\") pod \"multus-kxvqp\" (UID: \"ee639a0c-5f44-4091-a876-fe9109934f74\") " pod="openshift-multus/multus-kxvqp"
Oct 30 20:38:53.873509 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.873482    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lvnm\" (UniqueName: \"kubernetes.io/projected/9033c4e2-4586-4ec9-996b-21c412cfb8ba-kube-api-access-2lvnm\") pod \"periodic-gathering-4f444-dttkw\" (UID: \"9033c4e2-4586-4ec9-996b-21c412cfb8ba\") " pod="openshift-insights/periodic-gathering-4f444-dttkw"
Oct 30 20:38:53.873651 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.873529    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c3967ccc-990a-42f5-845e-cbeec5e18be0-var-lib-openvswitch\") pod \"ovnkube-node-mt2kl\" (UID: \"c3967ccc-990a-42f5-845e-cbeec5e18be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-mt2kl"
Oct 30 20:38:53.873651 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.873563    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ee639a0c-5f44-4091-a876-fe9109934f74-cnibin\") pod \"multus-kxvqp\" (UID: \"ee639a0c-5f44-4091-a876-fe9109934f74\") " pod="openshift-multus/multus-kxvqp"
Oct 30 20:38:53.873651 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.873592    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d5907106-812a-440a-b47d-f1889569d3b0-lib-modules\") pod \"tuned-pllmn\" (UID: \"d5907106-812a-440a-b47d-f1889569d3b0\") " pod="openshift-cluster-node-tuning-operator/tuned-pllmn"
Oct 30 20:38:53.873651 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.873610    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ee639a0c-5f44-4091-a876-fe9109934f74-host-var-lib-kubelet\") pod \"multus-kxvqp\" (UID: \"ee639a0c-5f44-4091-a876-fe9109934f74\") " pod="openshift-multus/multus-kxvqp"
Oct 30 20:38:53.873838 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.873662    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/30561fcc-3184-4116-b179-18941ff26a20-metrics-tls\") pod \"dns-default-fblmm\" (UID: \"30561fcc-3184-4116-b179-18941ff26a20\") " pod="openshift-dns/dns-default-fblmm"
Oct 30 20:38:53.873838 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.873699    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/451ecbeb-c188-4e49-8e4e-08af1051ae0f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lkn2k\" (UID: \"451ecbeb-c188-4e49-8e4e-08af1051ae0f\") " pod="openshift-insights/insights-runtime-extractor-lkn2k"
Oct 30 20:38:53.873838 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.873742    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c3967ccc-990a-42f5-845e-cbeec5e18be0-systemd-units\") pod \"ovnkube-node-mt2kl\" (UID: \"c3967ccc-990a-42f5-845e-cbeec5e18be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-mt2kl"
Oct 30 20:38:53.873838 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.873777    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c3967ccc-990a-42f5-845e-cbeec5e18be0-log-socket\") pod \"ovnkube-node-mt2kl\" (UID: \"c3967ccc-990a-42f5-845e-cbeec5e18be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-mt2kl"
Oct 30 20:38:53.873838 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.873828    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c3967ccc-990a-42f5-845e-cbeec5e18be0-host-run-ovn-kubernetes\") pod \"ovnkube-node-mt2kl\" (UID: \"c3967ccc-990a-42f5-845e-cbeec5e18be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-mt2kl"
Oct 30 20:38:53.874076 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.873955    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c3967ccc-990a-42f5-845e-cbeec5e18be0-log-socket\") pod \"ovnkube-node-mt2kl\" (UID: \"c3967ccc-990a-42f5-845e-cbeec5e18be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-mt2kl"
Oct 30 20:38:53.874076 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.873998    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ee639a0c-5f44-4091-a876-fe9109934f74-host-run-k8s-cni-cncf-io\") pod \"multus-kxvqp\" (UID: \"ee639a0c-5f44-4091-a876-fe9109934f74\") " pod="openshift-multus/multus-kxvqp"
Oct 30 20:38:53.874076 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.874033    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d5907106-812a-440a-b47d-f1889569d3b0-etc-sysconfig\") pod \"tuned-pllmn\" (UID: \"d5907106-812a-440a-b47d-f1889569d3b0\") " pod="openshift-cluster-node-tuning-operator/tuned-pllmn"
Oct 30 20:38:53.874076 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.874063    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d5907106-812a-440a-b47d-f1889569d3b0-etc-sysctl-d\") pod \"tuned-pllmn\" (UID: \"d5907106-812a-440a-b47d-f1889569d3b0\") " pod="openshift-cluster-node-tuning-operator/tuned-pllmn"
Oct 30 20:38:53.874249 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.874081    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c3967ccc-990a-42f5-845e-cbeec5e18be0-host-run-ovn-kubernetes\") pod \"ovnkube-node-mt2kl\" (UID: \"c3967ccc-990a-42f5-845e-cbeec5e18be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-mt2kl"
Oct 30 20:38:53.874249 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.874090    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6fdde40e-3330-4b84-a470-6a7e6a536dce-metrics-certs\") pod \"network-metrics-daemon-5bww9\" (UID: \"6fdde40e-3330-4b84-a470-6a7e6a536dce\") " pod="openshift-multus/network-metrics-daemon-5bww9"
Oct 30 20:38:53.874249 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.874123    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d5907106-812a-440a-b47d-f1889569d3b0-sys\") pod \"tuned-pllmn\" (UID: \"d5907106-812a-440a-b47d-f1889569d3b0\") " pod="openshift-cluster-node-tuning-operator/tuned-pllmn"
Oct 30 20:38:53.874249 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.874151    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a03cdbf5-91a2-41eb-8c1e-3c0f40da2632-dbus\") pod \"global-pull-secret-syncer-xkfsj\" (UID: \"a03cdbf5-91a2-41eb-8c1e-3c0f40da2632\") " pod="kube-system/global-pull-secret-syncer-xkfsj"
Oct 30 20:38:53.874249 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.874157    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ee639a0c-5f44-4091-a876-fe9109934f74-cnibin\") pod \"multus-kxvqp\" (UID: \"ee639a0c-5f44-4091-a876-fe9109934f74\") " pod="openshift-multus/multus-kxvqp"
Oct 30 20:38:53.874249 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.874191    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/76fb0a52-8bca-4610-9891-605df420cc7a-socket-dir\") pod \"aws-ebs-csi-driver-node-kxsjg\" (UID: \"76fb0a52-8bca-4610-9891-605df420cc7a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kxsjg"
Oct 30 20:38:53.874249 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.874198    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ee639a0c-5f44-4091-a876-fe9109934f74-host-var-lib-kubelet\") pod \"multus-kxvqp\" (UID: \"ee639a0c-5f44-4091-a876-fe9109934f74\") " pod="openshift-multus/multus-kxvqp"
Oct 30 20:38:53.874249 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.874226    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/1672e221-5127-44b8-bbc7-4d7b7ddc25c5-signing-cabundle\") pod \"service-ca-5fffc97bdf-f77w8\" (UID: \"1672e221-5127-44b8-bbc7-4d7b7ddc25c5\") " pod="openshift-service-ca/service-ca-5fffc97bdf-f77w8"
Oct 30 20:38:53.874579 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.874259    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ee639a0c-5f44-4091-a876-fe9109934f74-etc-kubernetes\") pod \"multus-kxvqp\" (UID: \"ee639a0c-5f44-4091-a876-fe9109934f74\") " pod="openshift-multus/multus-kxvqp"
Oct 30 20:38:53.874579 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.874294    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d5907106-812a-440a-b47d-f1889569d3b0-etc-modprobe-d\") pod \"tuned-pllmn\" (UID: \"d5907106-812a-440a-b47d-f1889569d3b0\") " pod="openshift-cluster-node-tuning-operator/tuned-pllmn"
Oct 30 20:38:53.874579 ip-10-0-131-125 kubenswrapper[4521]: E1030 20:38:53.874300    4521 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 20:38:53.874579 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.874314    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d5907106-812a-440a-b47d-f1889569d3b0-etc-sysctl-d\") pod \"tuned-pllmn\" (UID: \"d5907106-812a-440a-b47d-f1889569d3b0\") " pod="openshift-cluster-node-tuning-operator/tuned-pllmn"
Oct 30 20:38:53.874579 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.874351    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c6467dfc-7d9c-4ba8-ac2e-2ce19a74a0d4-hosts-file\") pod \"node-resolver-4kpcm\" (UID: \"c6467dfc-7d9c-4ba8-ac2e-2ce19a74a0d4\") " pod="openshift-dns/node-resolver-4kpcm"
Oct 30 20:38:53.874579 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.874389    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ee639a0c-5f44-4091-a876-fe9109934f74-host-run-k8s-cni-cncf-io\") pod \"multus-kxvqp\" (UID: \"ee639a0c-5f44-4091-a876-fe9109934f74\") " pod="openshift-multus/multus-kxvqp"
Oct 30 20:38:53.874579 ip-10-0-131-125 kubenswrapper[4521]: E1030 20:38:53.874417    4521 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30561fcc-3184-4116-b179-18941ff26a20-metrics-tls podName:30561fcc-3184-4116-b179-18941ff26a20 nodeName:}" failed. No retries permitted until 2026-10-30 20:38:54.374363168 +0000 UTC m=+1.719984401 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/30561fcc-3184-4116-b179-18941ff26a20-metrics-tls") pod "dns-default-fblmm" (UID: "30561fcc-3184-4116-b179-18941ff26a20") : secret "dns-default-metrics-tls" not found
Oct 30 20:38:53.874579 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.874419    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c6467dfc-7d9c-4ba8-ac2e-2ce19a74a0d4-hosts-file\") pod \"node-resolver-4kpcm\" (UID: \"c6467dfc-7d9c-4ba8-ac2e-2ce19a74a0d4\") " pod="openshift-dns/node-resolver-4kpcm"
Oct 30 20:38:53.874579 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.874438    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c3967ccc-990a-42f5-845e-cbeec5e18be0-host-slash\") pod \"ovnkube-node-mt2kl\" (UID: \"c3967ccc-990a-42f5-845e-cbeec5e18be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-mt2kl"
Oct 30 20:38:53.874579 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.874453    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d5907106-812a-440a-b47d-f1889569d3b0-etc-sysconfig\") pod \"tuned-pllmn\" (UID: \"d5907106-812a-440a-b47d-f1889569d3b0\") " pod="openshift-cluster-node-tuning-operator/tuned-pllmn"
Oct 30 20:38:53.874579 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.874480    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ee639a0c-5f44-4091-a876-fe9109934f74-host-run-netns\") pod \"multus-kxvqp\" (UID: \"ee639a0c-5f44-4091-a876-fe9109934f74\") " pod="openshift-multus/multus-kxvqp"
Oct 30 20:38:53.874579 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.874501    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c3967ccc-990a-42f5-845e-cbeec5e18be0-var-lib-openvswitch\") pod \"ovnkube-node-mt2kl\" (UID: \"c3967ccc-990a-42f5-845e-cbeec5e18be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-mt2kl"
Oct 30 20:38:53.874579 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.874511    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/76fb0a52-8bca-4610-9891-605df420cc7a-etc-selinux\") pod \"aws-ebs-csi-driver-node-kxsjg\" (UID: \"76fb0a52-8bca-4610-9891-605df420cc7a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kxsjg"
Oct 30 20:38:53.874579 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.874549    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/76fb0a52-8bca-4610-9891-605df420cc7a-sys-fs\") pod \"aws-ebs-csi-driver-node-kxsjg\" (UID: \"76fb0a52-8bca-4610-9891-605df420cc7a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kxsjg"
Oct 30 20:38:53.874579 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.874577    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/9033c4e2-4586-4ec9-996b-21c412cfb8ba-archives-path\") pod \"periodic-gathering-4f444-dttkw\" (UID: \"9033c4e2-4586-4ec9-996b-21c412cfb8ba\") " pod="openshift-insights/periodic-gathering-4f444-dttkw"
Oct 30 20:38:53.875207 ip-10-0-131-125 kubenswrapper[4521]: E1030 20:38:53.874606    4521 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 20:38:53.875207 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.874608    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9033c4e2-4586-4ec9-996b-21c412cfb8ba-service-ca-bundle\") pod \"periodic-gathering-4f444-dttkw\" (UID: \"9033c4e2-4586-4ec9-996b-21c412cfb8ba\") " pod="openshift-insights/periodic-gathering-4f444-dttkw"
Oct 30 20:38:53.875207 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.874641    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f831b1bf-4a1f-48b7-8242-85c063825277-host-slash\") pod \"iptables-alerter-9p5zk\" (UID: \"f831b1bf-4a1f-48b7-8242-85c063825277\") " pod="openshift-network-operator/iptables-alerter-9p5zk"
Oct 30 20:38:53.875207 ip-10-0-131-125 kubenswrapper[4521]: E1030 20:38:53.874677    4521 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6fdde40e-3330-4b84-a470-6a7e6a536dce-metrics-certs podName:6fdde40e-3330-4b84-a470-6a7e6a536dce nodeName:}" failed. No retries permitted until 2026-10-30 20:38:54.374649753 +0000 UTC m=+1.720270987 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6fdde40e-3330-4b84-a470-6a7e6a536dce-metrics-certs") pod "network-metrics-daemon-5bww9" (UID: "6fdde40e-3330-4b84-a470-6a7e6a536dce") : secret "metrics-daemon-secret" not found
Oct 30 20:38:53.875207 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.874683    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f831b1bf-4a1f-48b7-8242-85c063825277-host-slash\") pod \"iptables-alerter-9p5zk\" (UID: \"f831b1bf-4a1f-48b7-8242-85c063825277\") " pod="openshift-network-operator/iptables-alerter-9p5zk"
Oct 30 20:38:53.875207 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.874704    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c3967ccc-990a-42f5-845e-cbeec5e18be0-host-slash\") pod \"ovnkube-node-mt2kl\" (UID: \"c3967ccc-990a-42f5-845e-cbeec5e18be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-mt2kl"
Oct 30 20:38:53.875207 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.874717    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c3967ccc-990a-42f5-845e-cbeec5e18be0-etc-openvswitch\") pod \"ovnkube-node-mt2kl\" (UID: \"c3967ccc-990a-42f5-845e-cbeec5e18be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-mt2kl"
Oct 30 20:38:53.875207 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.874752    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d5907106-812a-440a-b47d-f1889569d3b0-etc-sysctl-conf\") pod \"tuned-pllmn\" (UID: \"d5907106-812a-440a-b47d-f1889569d3b0\") " pod="openshift-cluster-node-tuning-operator/tuned-pllmn"
Oct 30 20:38:53.875207 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.874783    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/9033c4e2-4586-4ec9-996b-21c412cfb8ba-archives-path\") pod \"periodic-gathering-4f444-dttkw\" (UID: \"9033c4e2-4586-4ec9-996b-21c412cfb8ba\") " pod="openshift-insights/periodic-gathering-4f444-dttkw"
Oct 30 20:38:53.875207 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.874785    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/1672e221-5127-44b8-bbc7-4d7b7ddc25c5-signing-key\") pod \"service-ca-5fffc97bdf-f77w8\" (UID: \"1672e221-5127-44b8-bbc7-4d7b7ddc25c5\") " pod="openshift-service-ca/service-ca-5fffc97bdf-f77w8"
Oct 30 20:38:53.875207 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.874837    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d5907106-812a-440a-b47d-f1889569d3b0-var-lib-kubelet\") pod \"tuned-pllmn\" (UID: \"d5907106-812a-440a-b47d-f1889569d3b0\") " pod="openshift-cluster-node-tuning-operator/tuned-pllmn"
Oct 30 20:38:53.875207 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.874870    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a03cdbf5-91a2-41eb-8c1e-3c0f40da2632-kubelet-config\") pod \"global-pull-secret-syncer-xkfsj\" (UID: \"a03cdbf5-91a2-41eb-8c1e-3c0f40da2632\") " pod="kube-system/global-pull-secret-syncer-xkfsj"
Oct 30 20:38:53.875207 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.874872    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9033c4e2-4586-4ec9-996b-21c412cfb8ba-service-ca-bundle\") pod \"periodic-gathering-4f444-dttkw\" (UID: \"9033c4e2-4586-4ec9-996b-21c412cfb8ba\") " pod="openshift-insights/periodic-gathering-4f444-dttkw"
Oct 30 20:38:53.875207 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.874874    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a03cdbf5-91a2-41eb-8c1e-3c0f40da2632-dbus\") pod \"global-pull-secret-syncer-xkfsj\" (UID: \"a03cdbf5-91a2-41eb-8c1e-3c0f40da2632\") " pod="kube-system/global-pull-secret-syncer-xkfsj"
Oct 30 20:38:53.875207 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.874916    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ee639a0c-5f44-4091-a876-fe9109934f74-host-run-multus-certs\") pod \"multus-kxvqp\" (UID: \"ee639a0c-5f44-4091-a876-fe9109934f74\") " pod="openshift-multus/multus-kxvqp"
Oct 30 20:38:53.875207 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.874941    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d5907106-812a-440a-b47d-f1889569d3b0-run\") pod \"tuned-pllmn\" (UID: \"d5907106-812a-440a-b47d-f1889569d3b0\") " pod="openshift-cluster-node-tuning-operator/tuned-pllmn"
Oct 30 20:38:53.875207 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.874945    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/76fb0a52-8bca-4610-9891-605df420cc7a-socket-dir\") pod \"aws-ebs-csi-driver-node-kxsjg\" (UID: \"76fb0a52-8bca-4610-9891-605df420cc7a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kxsjg"
Oct 30 20:38:53.875207 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.874947    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ee639a0c-5f44-4091-a876-fe9109934f74-etc-kubernetes\") pod \"multus-kxvqp\" (UID: \"ee639a0c-5f44-4091-a876-fe9109934f74\") " pod="openshift-multus/multus-kxvqp"
Oct 30 20:38:53.875207 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.874482    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c3967ccc-990a-42f5-845e-cbeec5e18be0-systemd-units\") pod \"ovnkube-node-mt2kl\" (UID: \"c3967ccc-990a-42f5-845e-cbeec5e18be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-mt2kl"
Oct 30 20:38:53.875207 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.874971    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e584a74f-f82c-4a3f-81d6-0b93601980dd-cnibin\") pod \"multus-additional-cni-plugins-p5hfb\" (UID: \"e584a74f-f82c-4a3f-81d6-0b93601980dd\") " pod="openshift-multus/multus-additional-cni-plugins-p5hfb"
Oct 30 20:38:53.875207 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.875009    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/76fb0a52-8bca-4610-9891-605df420cc7a-etc-selinux\") pod \"aws-ebs-csi-driver-node-kxsjg\" (UID: \"76fb0a52-8bca-4610-9891-605df420cc7a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kxsjg"
Oct 30 20:38:53.875207 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.875020    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e584a74f-f82c-4a3f-81d6-0b93601980dd-cnibin\") pod \"multus-additional-cni-plugins-p5hfb\" (UID: \"e584a74f-f82c-4a3f-81d6-0b93601980dd\") " pod="openshift-multus/multus-additional-cni-plugins-p5hfb"
Oct 30 20:38:53.875207 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.875051    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/451ecbeb-c188-4e49-8e4e-08af1051ae0f-crio-socket\") pod \"insights-runtime-extractor-lkn2k\" (UID: \"451ecbeb-c188-4e49-8e4e-08af1051ae0f\") " pod="openshift-insights/insights-runtime-extractor-lkn2k"
Oct 30 20:38:53.875207 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.875074    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c3967ccc-990a-42f5-845e-cbeec5e18be0-etc-openvswitch\") pod \"ovnkube-node-mt2kl\" (UID: \"c3967ccc-990a-42f5-845e-cbeec5e18be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-mt2kl"
Oct 30 20:38:53.875207 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.875083    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c3967ccc-990a-42f5-845e-cbeec5e18be0-node-log\") pod \"ovnkube-node-mt2kl\" (UID: \"c3967ccc-990a-42f5-845e-cbeec5e18be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-mt2kl"
Oct 30 20:38:53.875207 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.875117    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ee639a0c-5f44-4091-a876-fe9109934f74-multus-socket-dir-parent\") pod \"multus-kxvqp\" (UID: \"ee639a0c-5f44-4091-a876-fe9109934f74\") " pod="openshift-multus/multus-kxvqp"
Oct 30 20:38:53.875207 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.875145    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/76fb0a52-8bca-4610-9891-605df420cc7a-device-dir\") pod \"aws-ebs-csi-driver-node-kxsjg\" (UID: \"76fb0a52-8bca-4610-9891-605df420cc7a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kxsjg"
Oct 30 20:38:53.875207 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.875173    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c3967ccc-990a-42f5-845e-cbeec5e18be0-host-cni-netd\") pod \"ovnkube-node-mt2kl\" (UID: \"c3967ccc-990a-42f5-845e-cbeec5e18be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-mt2kl"
Oct 30 20:38:53.875207 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.875202    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ee639a0c-5f44-4091-a876-fe9109934f74-system-cni-dir\") pod \"multus-kxvqp\" (UID: \"ee639a0c-5f44-4091-a876-fe9109934f74\") " pod="openshift-multus/multus-kxvqp"
Oct 30 20:38:53.876425 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.875227    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d5907106-812a-440a-b47d-f1889569d3b0-etc-sysctl-conf\") pod \"tuned-pllmn\" (UID: \"d5907106-812a-440a-b47d-f1889569d3b0\") " pod="openshift-cluster-node-tuning-operator/tuned-pllmn"
Oct 30 20:38:53.876425 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.875231    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ee639a0c-5f44-4091-a876-fe9109934f74-host-var-lib-cni-multus\") pod \"multus-kxvqp\" (UID: \"ee639a0c-5f44-4091-a876-fe9109934f74\") " pod="openshift-multus/multus-kxvqp"
Oct 30 20:38:53.876425 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.875302    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ee639a0c-5f44-4091-a876-fe9109934f74-multus-socket-dir-parent\") pod \"multus-kxvqp\" (UID: \"ee639a0c-5f44-4091-a876-fe9109934f74\") " pod="openshift-multus/multus-kxvqp"
Oct 30 20:38:53.876425 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.875353    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d5907106-812a-440a-b47d-f1889569d3b0-etc-kubernetes\") pod \"tuned-pllmn\" (UID: \"d5907106-812a-440a-b47d-f1889569d3b0\") " pod="openshift-cluster-node-tuning-operator/tuned-pllmn"
Oct 30 20:38:53.876425 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.875385    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e584a74f-f82c-4a3f-81d6-0b93601980dd-system-cni-dir\") pod \"multus-additional-cni-plugins-p5hfb\" (UID: \"e584a74f-f82c-4a3f-81d6-0b93601980dd\") " pod="openshift-multus/multus-additional-cni-plugins-p5hfb"
Oct 30 20:38:53.876425 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.875416    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c3967ccc-990a-42f5-845e-cbeec5e18be0-host-run-netns\") pod \"ovnkube-node-mt2kl\" (UID: \"c3967ccc-990a-42f5-845e-cbeec5e18be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-mt2kl"
Oct 30 20:38:53.876425 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.875443    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ee639a0c-5f44-4091-a876-fe9109934f74-multus-conf-dir\") pod \"multus-kxvqp\" (UID: \"ee639a0c-5f44-4091-a876-fe9109934f74\") " pod="openshift-multus/multus-kxvqp"
Oct 30 20:38:53.876425 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.875466    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d5907106-812a-440a-b47d-f1889569d3b0-etc-systemd\") pod \"tuned-pllmn\" (UID: \"d5907106-812a-440a-b47d-f1889569d3b0\") " pod="openshift-cluster-node-tuning-operator/tuned-pllmn"
Oct 30 20:38:53.876425 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.875492    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d5907106-812a-440a-b47d-f1889569d3b0-host\") pod \"tuned-pllmn\" (UID: \"d5907106-812a-440a-b47d-f1889569d3b0\") " pod="openshift-cluster-node-tuning-operator/tuned-pllmn"
Oct 30 20:38:53.876425 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.875522    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e584a74f-f82c-4a3f-81d6-0b93601980dd-tuning-conf-dir\") pod \"multus-additional-cni-plugins-p5hfb\" (UID: \"e584a74f-f82c-4a3f-81d6-0b93601980dd\") " pod="openshift-multus/multus-additional-cni-plugins-p5hfb"
Oct 30 20:38:53.876425 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.875550    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/76fb0a52-8bca-4610-9891-605df420cc7a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kxsjg\" (UID: \"76fb0a52-8bca-4610-9891-605df420cc7a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kxsjg"
Oct 30 20:38:53.876425 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.875582    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/76fb0a52-8bca-4610-9891-605df420cc7a-registration-dir\") pod \"aws-ebs-csi-driver-node-kxsjg\" (UID: \"76fb0a52-8bca-4610-9891-605df420cc7a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kxsjg"
Oct 30 20:38:53.876425 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.875606    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c3967ccc-990a-42f5-845e-cbeec5e18be0-host-kubelet\") pod \"ovnkube-node-mt2kl\" (UID: \"c3967ccc-990a-42f5-845e-cbeec5e18be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-mt2kl"
Oct 30 20:38:53.876425 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.875648    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c3967ccc-990a-42f5-845e-cbeec5e18be0-run-systemd\") pod \"ovnkube-node-mt2kl\" (UID: \"c3967ccc-990a-42f5-845e-cbeec5e18be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-mt2kl"
Oct 30 20:38:53.876425 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.875721    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d5907106-812a-440a-b47d-f1889569d3b0-etc-modprobe-d\") pod \"tuned-pllmn\" (UID: \"d5907106-812a-440a-b47d-f1889569d3b0\") " pod="openshift-cluster-node-tuning-operator/tuned-pllmn"
Oct 30 20:38:53.876425 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.875739    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c3967ccc-990a-42f5-845e-cbeec5e18be0-run-systemd\") pod \"ovnkube-node-mt2kl\" (UID: \"c3967ccc-990a-42f5-845e-cbeec5e18be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-mt2kl"
Oct 30 20:38:53.876425 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.875740    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d5907106-812a-440a-b47d-f1889569d3b0-sys\") pod \"tuned-pllmn\" (UID: \"d5907106-812a-440a-b47d-f1889569d3b0\") " pod="openshift-cluster-node-tuning-operator/tuned-pllmn"
Oct 30 20:38:53.876425 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.875766    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/1672e221-5127-44b8-bbc7-4d7b7ddc25c5-signing-key\") pod \"service-ca-5fffc97bdf-f77w8\" (UID: \"1672e221-5127-44b8-bbc7-4d7b7ddc25c5\") " pod="openshift-service-ca/service-ca-5fffc97bdf-f77w8"
Oct 30 20:38:53.876425 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.875807    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d5907106-812a-440a-b47d-f1889569d3b0-var-lib-kubelet\") pod \"tuned-pllmn\" (UID: \"d5907106-812a-440a-b47d-f1889569d3b0\") " pod="openshift-cluster-node-tuning-operator/tuned-pllmn"
Oct 30 20:38:53.876425 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.875810    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ee639a0c-5f44-4091-a876-fe9109934f74-multus-conf-dir\") pod \"multus-kxvqp\" (UID: \"ee639a0c-5f44-4091-a876-fe9109934f74\") " pod="openshift-multus/multus-kxvqp"
Oct 30 20:38:53.876425 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.875818    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ee639a0c-5f44-4091-a876-fe9109934f74-system-cni-dir\") pod \"multus-kxvqp\" (UID: \"ee639a0c-5f44-4091-a876-fe9109934f74\") " pod="openshift-multus/multus-kxvqp"
Oct 30 20:38:53.876425 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.875854    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e584a74f-f82c-4a3f-81d6-0b93601980dd-system-cni-dir\") pod \"multus-additional-cni-plugins-p5hfb\" (UID: \"e584a74f-f82c-4a3f-81d6-0b93601980dd\") " pod="openshift-multus/multus-additional-cni-plugins-p5hfb"
Oct 30 20:38:53.876425 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.875859    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d5907106-812a-440a-b47d-f1889569d3b0-etc-kubernetes\") pod \"tuned-pllmn\" (UID: \"d5907106-812a-440a-b47d-f1889569d3b0\") " pod="openshift-cluster-node-tuning-operator/tuned-pllmn"
Oct 30 20:38:53.876425 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.875857    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c3967ccc-990a-42f5-845e-cbeec5e18be0-node-log\") pod \"ovnkube-node-mt2kl\" (UID: \"c3967ccc-990a-42f5-845e-cbeec5e18be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-mt2kl"
Oct 30 20:38:53.876425 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.875875    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/76fb0a52-8bca-4610-9891-605df420cc7a-registration-dir\") pod \"aws-ebs-csi-driver-node-kxsjg\" (UID: \"76fb0a52-8bca-4610-9891-605df420cc7a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kxsjg"
Oct 30 20:38:53.876425 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.875927    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ee639a0c-5f44-4091-a876-fe9109934f74-host-var-lib-cni-multus\") pod \"multus-kxvqp\" (UID: \"ee639a0c-5f44-4091-a876-fe9109934f74\") " pod="openshift-multus/multus-kxvqp"
Oct 30 20:38:53.876425 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.875936    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a03cdbf5-91a2-41eb-8c1e-3c0f40da2632-kubelet-config\") pod \"global-pull-secret-syncer-xkfsj\" (UID: \"a03cdbf5-91a2-41eb-8c1e-3c0f40da2632\") " pod="kube-system/global-pull-secret-syncer-xkfsj"
Oct 30 20:38:53.876425 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.875996    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/76fb0a52-8bca-4610-9891-605df420cc7a-sys-fs\") pod \"aws-ebs-csi-driver-node-kxsjg\" (UID: \"76fb0a52-8bca-4610-9891-605df420cc7a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kxsjg"
Oct 30 20:38:53.876425 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.875996    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ee639a0c-5f44-4091-a876-fe9109934f74-host-run-netns\") pod \"multus-kxvqp\" (UID: \"ee639a0c-5f44-4091-a876-fe9109934f74\") " pod="openshift-multus/multus-kxvqp"
Oct 30 20:38:53.876425 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.876126    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c3967ccc-990a-42f5-845e-cbeec5e18be0-host-run-netns\") pod \"ovnkube-node-mt2kl\" (UID: \"c3967ccc-990a-42f5-845e-cbeec5e18be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-mt2kl"
Oct 30 20:38:53.876425 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.876178    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c3967ccc-990a-42f5-845e-cbeec5e18be0-host-cni-netd\") pod \"ovnkube-node-mt2kl\" (UID: \"c3967ccc-990a-42f5-845e-cbeec5e18be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-mt2kl"
Oct 30 20:38:53.876425 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.876198    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/1672e221-5127-44b8-bbc7-4d7b7ddc25c5-signing-cabundle\") pod \"service-ca-5fffc97bdf-f77w8\" (UID: \"1672e221-5127-44b8-bbc7-4d7b7ddc25c5\") " pod="openshift-service-ca/service-ca-5fffc97bdf-f77w8"
Oct 30 20:38:53.876425 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.876239    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d5907106-812a-440a-b47d-f1889569d3b0-etc-systemd\") pod \"tuned-pllmn\" (UID: \"d5907106-812a-440a-b47d-f1889569d3b0\") " pod="openshift-cluster-node-tuning-operator/tuned-pllmn"
Oct 30 20:38:53.876425 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.876280    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e584a74f-f82c-4a3f-81d6-0b93601980dd-tuning-conf-dir\") pod \"multus-additional-cni-plugins-p5hfb\" (UID: \"e584a74f-f82c-4a3f-81d6-0b93601980dd\") " pod="openshift-multus/multus-additional-cni-plugins-p5hfb"
Oct 30 20:38:53.876425 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.876340    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/76fb0a52-8bca-4610-9891-605df420cc7a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kxsjg\" (UID: \"76fb0a52-8bca-4610-9891-605df420cc7a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kxsjg"
Oct 30 20:38:53.876425 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.876382    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c3967ccc-990a-42f5-845e-cbeec5e18be0-host-kubelet\") pod \"ovnkube-node-mt2kl\" (UID: \"c3967ccc-990a-42f5-845e-cbeec5e18be0\") " pod="openshift-ovn-kubernetes/ovnkube-node-mt2kl"
Oct 30 20:38:53.876425 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.876445    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d5907106-812a-440a-b47d-f1889569d3b0-run\") pod \"tuned-pllmn\" (UID: \"d5907106-812a-440a-b47d-f1889569d3b0\") " pod="openshift-cluster-node-tuning-operator/tuned-pllmn"
Oct 30 20:38:53.877797 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.876483    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ee639a0c-5f44-4091-a876-fe9109934f74-host-run-multus-certs\") pod \"multus-kxvqp\" (UID: \"ee639a0c-5f44-4091-a876-fe9109934f74\") " pod="openshift-multus/multus-kxvqp"
Oct 30 20:38:53.877797 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.876529    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d5907106-812a-440a-b47d-f1889569d3b0-host\") pod \"tuned-pllmn\" (UID: \"d5907106-812a-440a-b47d-f1889569d3b0\") " pod="openshift-cluster-node-tuning-operator/tuned-pllmn"
Oct 30 20:38:53.877797 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.876559    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/76fb0a52-8bca-4610-9891-605df420cc7a-device-dir\") pod \"aws-ebs-csi-driver-node-kxsjg\" (UID: \"76fb0a52-8bca-4610-9891-605df420cc7a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kxsjg"
Oct 30 20:38:53.883720 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.883698    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26z6g\" (UniqueName: \"kubernetes.io/projected/451ecbeb-c188-4e49-8e4e-08af1051ae0f-kube-api-access-26z6g\") pod \"insights-runtime-extractor-lkn2k\" (UID: \"451ecbeb-c188-4e49-8e4e-08af1051ae0f\") " pod="openshift-insights/insights-runtime-extractor-lkn2k"
Oct 30 20:38:53.883774 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.883715    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lvnm\" (UniqueName: \"kubernetes.io/projected/9033c4e2-4586-4ec9-996b-21c412cfb8ba-kube-api-access-2lvnm\") pod \"periodic-gathering-4f444-dttkw\" (UID: \"9033c4e2-4586-4ec9-996b-21c412cfb8ba\") " pod="openshift-insights/periodic-gathering-4f444-dttkw"
Oct 30 20:38:53.883865 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.883848    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8ggd\" (UniqueName: \"kubernetes.io/projected/1672e221-5127-44b8-bbc7-4d7b7ddc25c5-kube-api-access-r8ggd\") pod \"service-ca-5fffc97bdf-f77w8\" (UID: \"1672e221-5127-44b8-bbc7-4d7b7ddc25c5\") " pod="openshift-service-ca/service-ca-5fffc97bdf-f77w8"
Oct 30 20:38:53.976889 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.976857    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/451ecbeb-c188-4e49-8e4e-08af1051ae0f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lkn2k\" (UID: \"451ecbeb-c188-4e49-8e4e-08af1051ae0f\") " pod="openshift-insights/insights-runtime-extractor-lkn2k"
Oct 30 20:38:53.977047 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.976901    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/451ecbeb-c188-4e49-8e4e-08af1051ae0f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lkn2k\" (UID: \"451ecbeb-c188-4e49-8e4e-08af1051ae0f\") " pod="openshift-insights/insights-runtime-extractor-lkn2k"
Oct 30 20:38:53.977047 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.976943    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/451ecbeb-c188-4e49-8e4e-08af1051ae0f-crio-socket\") pod \"insights-runtime-extractor-lkn2k\" (UID: \"451ecbeb-c188-4e49-8e4e-08af1051ae0f\") " pod="openshift-insights/insights-runtime-extractor-lkn2k"
Oct 30 20:38:53.977047 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.977019    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/451ecbeb-c188-4e49-8e4e-08af1051ae0f-crio-socket\") pod \"insights-runtime-extractor-lkn2k\" (UID: \"451ecbeb-c188-4e49-8e4e-08af1051ae0f\") " pod="openshift-insights/insights-runtime-extractor-lkn2k"
Oct 30 20:38:53.977158 ip-10-0-131-125 kubenswrapper[4521]: E1030 20:38:53.977070    4521 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 20:38:53.977158 ip-10-0-131-125 kubenswrapper[4521]: E1030 20:38:53.977140    4521 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/451ecbeb-c188-4e49-8e4e-08af1051ae0f-insights-runtime-extractor-tls podName:451ecbeb-c188-4e49-8e4e-08af1051ae0f nodeName:}" failed. No retries permitted until 2026-10-30 20:38:54.477125031 +0000 UTC m=+1.822746247 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/451ecbeb-c188-4e49-8e4e-08af1051ae0f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lkn2k" (UID: "451ecbeb-c188-4e49-8e4e-08af1051ae0f") : secret "insights-runtime-extractor-tls" not found
Oct 30 20:38:53.977401 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:53.977384    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/451ecbeb-c188-4e49-8e4e-08af1051ae0f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lkn2k\" (UID: \"451ecbeb-c188-4e49-8e4e-08af1051ae0f\") " pod="openshift-insights/insights-runtime-extractor-lkn2k"
Oct 30 20:38:54.063665 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:54.063647    4521 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 20:38:54.077810 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:54.077777    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-f4cch"
Oct 30 20:38:54.324926 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:54.324856    4521 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-4kpcm_c6467dfc-7d9c-4ba8-ac2e-2ce19a74a0d4/dns-node-resolver/0.log"
Oct 30 20:38:54.380709 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:54.380670    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/30561fcc-3184-4116-b179-18941ff26a20-metrics-tls\") pod \"dns-default-fblmm\" (UID: \"30561fcc-3184-4116-b179-18941ff26a20\") " pod="openshift-dns/dns-default-fblmm"
Oct 30 20:38:54.380875 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:54.380724    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6fdde40e-3330-4b84-a470-6a7e6a536dce-metrics-certs\") pod \"network-metrics-daemon-5bww9\" (UID: \"6fdde40e-3330-4b84-a470-6a7e6a536dce\") " pod="openshift-multus/network-metrics-daemon-5bww9"
Oct 30 20:38:54.380875 ip-10-0-131-125 kubenswrapper[4521]: E1030 20:38:54.380819    4521 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 20:38:54.380875 ip-10-0-131-125 kubenswrapper[4521]: E1030 20:38:54.380864    4521 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 20:38:54.381015 ip-10-0-131-125 kubenswrapper[4521]: E1030 20:38:54.380882    4521 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30561fcc-3184-4116-b179-18941ff26a20-metrics-tls podName:30561fcc-3184-4116-b179-18941ff26a20 nodeName:}" failed. No retries permitted until 2026-10-30 20:38:55.380867245 +0000 UTC m=+2.726488461 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/30561fcc-3184-4116-b179-18941ff26a20-metrics-tls") pod "dns-default-fblmm" (UID: "30561fcc-3184-4116-b179-18941ff26a20") : secret "dns-default-metrics-tls" not found
Oct 30 20:38:54.381015 ip-10-0-131-125 kubenswrapper[4521]: E1030 20:38:54.380913    4521 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6fdde40e-3330-4b84-a470-6a7e6a536dce-metrics-certs podName:6fdde40e-3330-4b84-a470-6a7e6a536dce nodeName:}" failed. No retries permitted until 2026-10-30 20:38:55.380897731 +0000 UTC m=+2.726518961 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6fdde40e-3330-4b84-a470-6a7e6a536dce-metrics-certs") pod "network-metrics-daemon-5bww9" (UID: "6fdde40e-3330-4b84-a470-6a7e6a536dce") : secret "metrics-daemon-secret" not found
Oct 30 20:38:54.481770 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:54.481729    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/451ecbeb-c188-4e49-8e4e-08af1051ae0f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lkn2k\" (UID: \"451ecbeb-c188-4e49-8e4e-08af1051ae0f\") " pod="openshift-insights/insights-runtime-extractor-lkn2k"
Oct 30 20:38:54.481917 ip-10-0-131-125 kubenswrapper[4521]: E1030 20:38:54.481882    4521 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 20:38:54.481960 ip-10-0-131-125 kubenswrapper[4521]: E1030 20:38:54.481950    4521 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/451ecbeb-c188-4e49-8e4e-08af1051ae0f-insights-runtime-extractor-tls podName:451ecbeb-c188-4e49-8e4e-08af1051ae0f nodeName:}" failed. No retries permitted until 2026-10-30 20:38:55.481936067 +0000 UTC m=+2.827557283 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/451ecbeb-c188-4e49-8e4e-08af1051ae0f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lkn2k" (UID: "451ecbeb-c188-4e49-8e4e-08af1051ae0f") : secret "insights-runtime-extractor-tls" not found
Oct 30 20:38:54.826488 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:54.826448    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-f77w8" event={"ID":"1672e221-5127-44b8-bbc7-4d7b7ddc25c5","Type":"ContainerStarted","Data":"bf839a8f3ec354d140e46e459794f8d27bff5e63c00649a05f72a04160b4781d"}
Oct 30 20:38:55.325931 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:55.325899    4521 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-2kdx9_880d0493-e39d-4d9c-b58b-aa48c33aeaef/node-ca/0.log"
Oct 30 20:38:55.388423 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:55.388386    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/30561fcc-3184-4116-b179-18941ff26a20-metrics-tls\") pod \"dns-default-fblmm\" (UID: \"30561fcc-3184-4116-b179-18941ff26a20\") " pod="openshift-dns/dns-default-fblmm"
Oct 30 20:38:55.388628 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:55.388444    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6fdde40e-3330-4b84-a470-6a7e6a536dce-metrics-certs\") pod \"network-metrics-daemon-5bww9\" (UID: \"6fdde40e-3330-4b84-a470-6a7e6a536dce\") " pod="openshift-multus/network-metrics-daemon-5bww9"
Oct 30 20:38:55.388628 ip-10-0-131-125 kubenswrapper[4521]: E1030 20:38:55.388561    4521 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 20:38:55.388728 ip-10-0-131-125 kubenswrapper[4521]: E1030 20:38:55.388635    4521 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30561fcc-3184-4116-b179-18941ff26a20-metrics-tls podName:30561fcc-3184-4116-b179-18941ff26a20 nodeName:}" failed. No retries permitted until 2026-10-30 20:38:57.388614425 +0000 UTC m=+4.734235655 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/30561fcc-3184-4116-b179-18941ff26a20-metrics-tls") pod "dns-default-fblmm" (UID: "30561fcc-3184-4116-b179-18941ff26a20") : secret "dns-default-metrics-tls" not found
Oct 30 20:38:55.388728 ip-10-0-131-125 kubenswrapper[4521]: E1030 20:38:55.388565    4521 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 20:38:55.388728 ip-10-0-131-125 kubenswrapper[4521]: E1030 20:38:55.388707    4521 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6fdde40e-3330-4b84-a470-6a7e6a536dce-metrics-certs podName:6fdde40e-3330-4b84-a470-6a7e6a536dce nodeName:}" failed. No retries permitted until 2026-10-30 20:38:57.388691507 +0000 UTC m=+4.734312727 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6fdde40e-3330-4b84-a470-6a7e6a536dce-metrics-certs") pod "network-metrics-daemon-5bww9" (UID: "6fdde40e-3330-4b84-a470-6a7e6a536dce") : secret "metrics-daemon-secret" not found
Oct 30 20:38:55.489532 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:55.489493    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/451ecbeb-c188-4e49-8e4e-08af1051ae0f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lkn2k\" (UID: \"451ecbeb-c188-4e49-8e4e-08af1051ae0f\") " pod="openshift-insights/insights-runtime-extractor-lkn2k"
Oct 30 20:38:55.489712 ip-10-0-131-125 kubenswrapper[4521]: E1030 20:38:55.489656    4521 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 20:38:55.489763 ip-10-0-131-125 kubenswrapper[4521]: E1030 20:38:55.489728    4521 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/451ecbeb-c188-4e49-8e4e-08af1051ae0f-insights-runtime-extractor-tls podName:451ecbeb-c188-4e49-8e4e-08af1051ae0f nodeName:}" failed. No retries permitted until 2026-10-30 20:38:57.489710354 +0000 UTC m=+4.835331578 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/451ecbeb-c188-4e49-8e4e-08af1051ae0f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lkn2k" (UID: "451ecbeb-c188-4e49-8e4e-08af1051ae0f") : secret "insights-runtime-extractor-tls" not found
Oct 30 20:38:56.325741 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:56.325703    4521 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-q2wtm_747c15be-3782-4065-a6dc-73d9fe7a4b4f/serve-healthcheck-canary/0.log"
Oct 30 20:38:56.725911 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:56.725879    4521 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-wqbfm_932cdb01-d506-4c20-9b4d-902da81e1f3c/migrator/0.log"
Oct 30 20:38:57.402608 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:57.402571    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/30561fcc-3184-4116-b179-18941ff26a20-metrics-tls\") pod \"dns-default-fblmm\" (UID: \"30561fcc-3184-4116-b179-18941ff26a20\") " pod="openshift-dns/dns-default-fblmm"
Oct 30 20:38:57.403013 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:57.402621    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6fdde40e-3330-4b84-a470-6a7e6a536dce-metrics-certs\") pod \"network-metrics-daemon-5bww9\" (UID: \"6fdde40e-3330-4b84-a470-6a7e6a536dce\") " pod="openshift-multus/network-metrics-daemon-5bww9"
Oct 30 20:38:57.403013 ip-10-0-131-125 kubenswrapper[4521]: E1030 20:38:57.402735    4521 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 20:38:57.403013 ip-10-0-131-125 kubenswrapper[4521]: E1030 20:38:57.402793    4521 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6fdde40e-3330-4b84-a470-6a7e6a536dce-metrics-certs podName:6fdde40e-3330-4b84-a470-6a7e6a536dce nodeName:}" failed. No retries permitted until 2026-10-30 20:39:01.402780482 +0000 UTC m=+8.748401697 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6fdde40e-3330-4b84-a470-6a7e6a536dce-metrics-certs") pod "network-metrics-daemon-5bww9" (UID: "6fdde40e-3330-4b84-a470-6a7e6a536dce") : secret "metrics-daemon-secret" not found
Oct 30 20:38:57.403133 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:57.403002    4521 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 20:38:57.405146 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:57.405118    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/30561fcc-3184-4116-b179-18941ff26a20-metrics-tls\") pod \"dns-default-fblmm\" (UID: \"30561fcc-3184-4116-b179-18941ff26a20\") " pod="openshift-dns/dns-default-fblmm"
Oct 30 20:38:57.503924 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:57.503886    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/451ecbeb-c188-4e49-8e4e-08af1051ae0f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lkn2k\" (UID: \"451ecbeb-c188-4e49-8e4e-08af1051ae0f\") " pod="openshift-insights/insights-runtime-extractor-lkn2k"
Oct 30 20:38:57.504119 ip-10-0-131-125 kubenswrapper[4521]: E1030 20:38:57.504061    4521 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 20:38:57.504183 ip-10-0-131-125 kubenswrapper[4521]: E1030 20:38:57.504142    4521 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/451ecbeb-c188-4e49-8e4e-08af1051ae0f-insights-runtime-extractor-tls podName:451ecbeb-c188-4e49-8e4e-08af1051ae0f nodeName:}" failed. No retries permitted until 2026-10-30 20:39:01.504122935 +0000 UTC m=+8.849744168 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/451ecbeb-c188-4e49-8e4e-08af1051ae0f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lkn2k" (UID: "451ecbeb-c188-4e49-8e4e-08af1051ae0f") : secret "insights-runtime-extractor-tls" not found
Oct 30 20:38:57.659510 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:57.659434    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-fblmm"
Oct 30 20:38:57.929215 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:57.929145    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-fblmm"]
Oct 30 20:38:57.933133 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:38:57.933104    4521 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30561fcc_3184_4116_b179_18941ff26a20.slice/crio-72bf293850e1822c0b6b3e5bfa56d37d18daf87145f99d598743c23817ad673b WatchSource:0}: Error finding container 72bf293850e1822c0b6b3e5bfa56d37d18daf87145f99d598743c23817ad673b: Status 404 returned error can't find the container with id 72bf293850e1822c0b6b3e5bfa56d37d18daf87145f99d598743c23817ad673b
Oct 30 20:38:58.506964 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:58.506933    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mt2kl"
Oct 30 20:38:58.840533 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:58.840456    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-fblmm" event={"ID":"30561fcc-3184-4116-b179-18941ff26a20","Type":"ContainerStarted","Data":"72bf293850e1822c0b6b3e5bfa56d37d18daf87145f99d598743c23817ad673b"}
Oct 30 20:38:59.845416 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:59.845310    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-fblmm" event={"ID":"30561fcc-3184-4116-b179-18941ff26a20","Type":"ContainerStarted","Data":"5d61b17967da4d50af3df4bd07facc07fca0b673ab786b70779c33a6c08c93a2"}
Oct 30 20:38:59.845416 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:59.845381    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-fblmm" event={"ID":"30561fcc-3184-4116-b179-18941ff26a20","Type":"ContainerStarted","Data":"7f8d1d7ff40a45f0920b11dd7a05a378e48e65d1f31410eb8f813d63b30fdbc9"}
Oct 30 20:38:59.845865 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:38:59.845473    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-fblmm"
Oct 30 20:39:01.434483 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:01.434440    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6fdde40e-3330-4b84-a470-6a7e6a536dce-metrics-certs\") pod \"network-metrics-daemon-5bww9\" (UID: \"6fdde40e-3330-4b84-a470-6a7e6a536dce\") " pod="openshift-multus/network-metrics-daemon-5bww9"
Oct 30 20:39:01.434993 ip-10-0-131-125 kubenswrapper[4521]: E1030 20:39:01.434607    4521 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 20:39:01.434993 ip-10-0-131-125 kubenswrapper[4521]: E1030 20:39:01.434694    4521 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6fdde40e-3330-4b84-a470-6a7e6a536dce-metrics-certs podName:6fdde40e-3330-4b84-a470-6a7e6a536dce nodeName:}" failed. No retries permitted until 2026-10-30 20:39:09.434673265 +0000 UTC m=+16.780294492 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6fdde40e-3330-4b84-a470-6a7e6a536dce-metrics-certs") pod "network-metrics-daemon-5bww9" (UID: "6fdde40e-3330-4b84-a470-6a7e6a536dce") : secret "metrics-daemon-secret" not found
Oct 30 20:39:01.535156 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:01.535111    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/451ecbeb-c188-4e49-8e4e-08af1051ae0f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lkn2k\" (UID: \"451ecbeb-c188-4e49-8e4e-08af1051ae0f\") " pod="openshift-insights/insights-runtime-extractor-lkn2k"
Oct 30 20:39:01.537464 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:01.537443    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/451ecbeb-c188-4e49-8e4e-08af1051ae0f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lkn2k\" (UID: \"451ecbeb-c188-4e49-8e4e-08af1051ae0f\") " pod="openshift-insights/insights-runtime-extractor-lkn2k"
Oct 30 20:39:01.562517 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:01.562473    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-lkn2k"
Oct 30 20:39:01.835640 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:01.835603    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-lkn2k"]
Oct 30 20:39:01.839078 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:39:01.839048    4521 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod451ecbeb_c188_4e49_8e4e_08af1051ae0f.slice/crio-ac93636bbbe5a3972804beb70eedd5798481d86a35e00c863e28be6363ac539a WatchSource:0}: Error finding container ac93636bbbe5a3972804beb70eedd5798481d86a35e00c863e28be6363ac539a: Status 404 returned error can't find the container with id ac93636bbbe5a3972804beb70eedd5798481d86a35e00c863e28be6363ac539a
Oct 30 20:39:01.853582 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:01.853552    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lkn2k" event={"ID":"451ecbeb-c188-4e49-8e4e-08af1051ae0f","Type":"ContainerStarted","Data":"ac93636bbbe5a3972804beb70eedd5798481d86a35e00c863e28be6363ac539a"}
Oct 30 20:39:02.858070 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:02.858033    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lkn2k" event={"ID":"451ecbeb-c188-4e49-8e4e-08af1051ae0f","Type":"ContainerStarted","Data":"987995b9107359d5e483dc37113a617086cb73237514b8a628f45834eb733e8c"}
Oct 30 20:39:02.858520 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:02.858092    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lkn2k" event={"ID":"451ecbeb-c188-4e49-8e4e-08af1051ae0f","Type":"ContainerStarted","Data":"46d4909da44fb27432a99a72fc7ca9b775ad196b5af0440140900659347cc88b"}
Oct 30 20:39:03.661618 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:03.661581    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mt2kl"
Oct 30 20:39:03.661856 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:03.661838    4521 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 20:39:03.814613 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:03.814576    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-qcrpl"
Oct 30 20:39:04.866525 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:04.866487    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lkn2k" event={"ID":"451ecbeb-c188-4e49-8e4e-08af1051ae0f","Type":"ContainerStarted","Data":"187f135ff80638de7aec09450776a3679b126c4038e87d20adfc302190252b24"}
Oct 30 20:39:08.444002 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:08.443963    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mt2kl"
Oct 30 20:39:09.019888 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:09.019850    4521 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7579767596-gmrh5"]
Oct 30 20:39:09.028594 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:09.028569    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7579767596-gmrh5"
Oct 30 20:39:09.044519 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:09.044483    4521 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Oct 30 20:39:09.045263 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:09.045239    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7579767596-gmrh5"]
Oct 30 20:39:09.053391 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:09.053371    4521 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Oct 30 20:39:09.053632 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:09.053610    4521 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Oct 30 20:39:09.053682 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:09.053622    4521 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Oct 30 20:39:09.054035 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:09.054019    4521 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-4f5dn\""
Oct 30 20:39:09.087119 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:09.087087    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ec3ad2a6-6025-4187-8bac-1ab4d7ea5de0-bound-sa-token\") pod \"image-registry-7579767596-gmrh5\" (UID: \"ec3ad2a6-6025-4187-8bac-1ab4d7ea5de0\") " pod="openshift-image-registry/image-registry-7579767596-gmrh5"
Oct 30 20:39:09.087251 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:09.087123    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4b6tp\" (UniqueName: \"kubernetes.io/projected/ec3ad2a6-6025-4187-8bac-1ab4d7ea5de0-kube-api-access-4b6tp\") pod \"image-registry-7579767596-gmrh5\" (UID: \"ec3ad2a6-6025-4187-8bac-1ab4d7ea5de0\") " pod="openshift-image-registry/image-registry-7579767596-gmrh5"
Oct 30 20:39:09.087251 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:09.087151    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ec3ad2a6-6025-4187-8bac-1ab4d7ea5de0-ca-trust-extracted\") pod \"image-registry-7579767596-gmrh5\" (UID: \"ec3ad2a6-6025-4187-8bac-1ab4d7ea5de0\") " pod="openshift-image-registry/image-registry-7579767596-gmrh5"
Oct 30 20:39:09.087374 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:09.087286    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ec3ad2a6-6025-4187-8bac-1ab4d7ea5de0-image-registry-private-configuration\") pod \"image-registry-7579767596-gmrh5\" (UID: \"ec3ad2a6-6025-4187-8bac-1ab4d7ea5de0\") " pod="openshift-image-registry/image-registry-7579767596-gmrh5"
Oct 30 20:39:09.087374 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:09.087332    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ec3ad2a6-6025-4187-8bac-1ab4d7ea5de0-trusted-ca\") pod \"image-registry-7579767596-gmrh5\" (UID: \"ec3ad2a6-6025-4187-8bac-1ab4d7ea5de0\") " pod="openshift-image-registry/image-registry-7579767596-gmrh5"
Oct 30 20:39:09.087451 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:09.087424    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ec3ad2a6-6025-4187-8bac-1ab4d7ea5de0-registry-tls\") pod \"image-registry-7579767596-gmrh5\" (UID: \"ec3ad2a6-6025-4187-8bac-1ab4d7ea5de0\") " pod="openshift-image-registry/image-registry-7579767596-gmrh5"
Oct 30 20:39:09.087486 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:09.087455    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ec3ad2a6-6025-4187-8bac-1ab4d7ea5de0-installation-pull-secrets\") pod \"image-registry-7579767596-gmrh5\" (UID: \"ec3ad2a6-6025-4187-8bac-1ab4d7ea5de0\") " pod="openshift-image-registry/image-registry-7579767596-gmrh5"
Oct 30 20:39:09.087538 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:09.087490    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ec3ad2a6-6025-4187-8bac-1ab4d7ea5de0-registry-certificates\") pod \"image-registry-7579767596-gmrh5\" (UID: \"ec3ad2a6-6025-4187-8bac-1ab4d7ea5de0\") " pod="openshift-image-registry/image-registry-7579767596-gmrh5"
Oct 30 20:39:09.188493 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:09.188446    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ec3ad2a6-6025-4187-8bac-1ab4d7ea5de0-image-registry-private-configuration\") pod \"image-registry-7579767596-gmrh5\" (UID: \"ec3ad2a6-6025-4187-8bac-1ab4d7ea5de0\") " pod="openshift-image-registry/image-registry-7579767596-gmrh5"
Oct 30 20:39:09.188493 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:09.188493    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ec3ad2a6-6025-4187-8bac-1ab4d7ea5de0-trusted-ca\") pod \"image-registry-7579767596-gmrh5\" (UID: \"ec3ad2a6-6025-4187-8bac-1ab4d7ea5de0\") " pod="openshift-image-registry/image-registry-7579767596-gmrh5"
Oct 30 20:39:09.188743 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:09.188677    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ec3ad2a6-6025-4187-8bac-1ab4d7ea5de0-registry-tls\") pod \"image-registry-7579767596-gmrh5\" (UID: \"ec3ad2a6-6025-4187-8bac-1ab4d7ea5de0\") " pod="openshift-image-registry/image-registry-7579767596-gmrh5"
Oct 30 20:39:09.188743 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:09.188716    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ec3ad2a6-6025-4187-8bac-1ab4d7ea5de0-installation-pull-secrets\") pod \"image-registry-7579767596-gmrh5\" (UID: \"ec3ad2a6-6025-4187-8bac-1ab4d7ea5de0\") " pod="openshift-image-registry/image-registry-7579767596-gmrh5"
Oct 30 20:39:09.188854 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:09.188760    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ec3ad2a6-6025-4187-8bac-1ab4d7ea5de0-registry-certificates\") pod \"image-registry-7579767596-gmrh5\" (UID: \"ec3ad2a6-6025-4187-8bac-1ab4d7ea5de0\") " pod="openshift-image-registry/image-registry-7579767596-gmrh5"
Oct 30 20:39:09.188854 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:09.188778    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ec3ad2a6-6025-4187-8bac-1ab4d7ea5de0-bound-sa-token\") pod \"image-registry-7579767596-gmrh5\" (UID: \"ec3ad2a6-6025-4187-8bac-1ab4d7ea5de0\") " pod="openshift-image-registry/image-registry-7579767596-gmrh5"
Oct 30 20:39:09.188854 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:09.188793    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4b6tp\" (UniqueName: \"kubernetes.io/projected/ec3ad2a6-6025-4187-8bac-1ab4d7ea5de0-kube-api-access-4b6tp\") pod \"image-registry-7579767596-gmrh5\" (UID: \"ec3ad2a6-6025-4187-8bac-1ab4d7ea5de0\") " pod="openshift-image-registry/image-registry-7579767596-gmrh5"
Oct 30 20:39:09.188854 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:09.188809    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ec3ad2a6-6025-4187-8bac-1ab4d7ea5de0-ca-trust-extracted\") pod \"image-registry-7579767596-gmrh5\" (UID: \"ec3ad2a6-6025-4187-8bac-1ab4d7ea5de0\") " pod="openshift-image-registry/image-registry-7579767596-gmrh5"
Oct 30 20:39:09.189253 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:09.189195    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ec3ad2a6-6025-4187-8bac-1ab4d7ea5de0-ca-trust-extracted\") pod \"image-registry-7579767596-gmrh5\" (UID: \"ec3ad2a6-6025-4187-8bac-1ab4d7ea5de0\") " pod="openshift-image-registry/image-registry-7579767596-gmrh5"
Oct 30 20:39:09.189552 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:09.189524    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ec3ad2a6-6025-4187-8bac-1ab4d7ea5de0-registry-certificates\") pod \"image-registry-7579767596-gmrh5\" (UID: \"ec3ad2a6-6025-4187-8bac-1ab4d7ea5de0\") " pod="openshift-image-registry/image-registry-7579767596-gmrh5"
Oct 30 20:39:09.189707 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:09.189690    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ec3ad2a6-6025-4187-8bac-1ab4d7ea5de0-trusted-ca\") pod \"image-registry-7579767596-gmrh5\" (UID: \"ec3ad2a6-6025-4187-8bac-1ab4d7ea5de0\") " pod="openshift-image-registry/image-registry-7579767596-gmrh5"
Oct 30 20:39:09.191124 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:09.191092    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ec3ad2a6-6025-4187-8bac-1ab4d7ea5de0-registry-tls\") pod \"image-registry-7579767596-gmrh5\" (UID: \"ec3ad2a6-6025-4187-8bac-1ab4d7ea5de0\") " pod="openshift-image-registry/image-registry-7579767596-gmrh5"
Oct 30 20:39:09.191216 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:09.191149    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ec3ad2a6-6025-4187-8bac-1ab4d7ea5de0-installation-pull-secrets\") pod \"image-registry-7579767596-gmrh5\" (UID: \"ec3ad2a6-6025-4187-8bac-1ab4d7ea5de0\") " pod="openshift-image-registry/image-registry-7579767596-gmrh5"
Oct 30 20:39:09.191263 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:09.191253    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ec3ad2a6-6025-4187-8bac-1ab4d7ea5de0-image-registry-private-configuration\") pod \"image-registry-7579767596-gmrh5\" (UID: \"ec3ad2a6-6025-4187-8bac-1ab4d7ea5de0\") " pod="openshift-image-registry/image-registry-7579767596-gmrh5"
Oct 30 20:39:09.201454 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:09.201431    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ec3ad2a6-6025-4187-8bac-1ab4d7ea5de0-bound-sa-token\") pod \"image-registry-7579767596-gmrh5\" (UID: \"ec3ad2a6-6025-4187-8bac-1ab4d7ea5de0\") " pod="openshift-image-registry/image-registry-7579767596-gmrh5"
Oct 30 20:39:09.201661 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:09.201644    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4b6tp\" (UniqueName: \"kubernetes.io/projected/ec3ad2a6-6025-4187-8bac-1ab4d7ea5de0-kube-api-access-4b6tp\") pod \"image-registry-7579767596-gmrh5\" (UID: \"ec3ad2a6-6025-4187-8bac-1ab4d7ea5de0\") " pod="openshift-image-registry/image-registry-7579767596-gmrh5"
Oct 30 20:39:09.343879 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:09.343796    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7579767596-gmrh5"
Oct 30 20:39:09.491977 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:09.491936    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6fdde40e-3330-4b84-a470-6a7e6a536dce-metrics-certs\") pod \"network-metrics-daemon-5bww9\" (UID: \"6fdde40e-3330-4b84-a470-6a7e6a536dce\") " pod="openshift-multus/network-metrics-daemon-5bww9"
Oct 30 20:39:09.494178 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:09.494158    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6fdde40e-3330-4b84-a470-6a7e6a536dce-metrics-certs\") pod \"network-metrics-daemon-5bww9\" (UID: \"6fdde40e-3330-4b84-a470-6a7e6a536dce\") " pod="openshift-multus/network-metrics-daemon-5bww9"
Oct 30 20:39:09.621063 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:09.620979    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7579767596-gmrh5"]
Oct 30 20:39:09.625145 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:39:09.625118    4521 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec3ad2a6_6025_4187_8bac_1ab4d7ea5de0.slice/crio-ea54096a62d88c198ac4d74024735e7c62a0eebaabd796ff99990fa7bb41c38d WatchSource:0}: Error finding container ea54096a62d88c198ac4d74024735e7c62a0eebaabd796ff99990fa7bb41c38d: Status 404 returned error can't find the container with id ea54096a62d88c198ac4d74024735e7c62a0eebaabd796ff99990fa7bb41c38d
Oct 30 20:39:09.653509 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:09.653482    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5bww9"
Oct 30 20:39:09.851437 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:09.851406    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-fblmm"
Oct 30 20:39:09.886006 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:09.885917    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7579767596-gmrh5" event={"ID":"ec3ad2a6-6025-4187-8bac-1ab4d7ea5de0","Type":"ContainerStarted","Data":"6f34b1c9f185fc562d399ff64bab5c4ec6288422adcdf00545b21cf69107eeaa"}
Oct 30 20:39:09.886006 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:09.885965    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7579767596-gmrh5" event={"ID":"ec3ad2a6-6025-4187-8bac-1ab4d7ea5de0","Type":"ContainerStarted","Data":"ea54096a62d88c198ac4d74024735e7c62a0eebaabd796ff99990fa7bb41c38d"}
Oct 30 20:39:09.886217 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:09.886059    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7579767596-gmrh5"
Oct 30 20:39:09.914900 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:09.914849    4521 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7579767596-gmrh5" podStartSLOduration=1.914835834 podStartE2EDuration="1.914835834s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:39:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:39:09.9139509 +0000 UTC m=+17.259572161" watchObservedRunningTime="2026-10-30 20:39:09.914835834 +0000 UTC m=+17.260457072"
Oct 30 20:39:09.929457 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:09.929427    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-5bww9"]
Oct 30 20:39:09.933838 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:39:09.933798    4521 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fdde40e_3330_4b84_a470_6a7e6a536dce.slice/crio-ed1ac5e4aa8f59cc6737bea82f74383d6ed8f9fb8a9e844e5f716aea9f6b90d0 WatchSource:0}: Error finding container ed1ac5e4aa8f59cc6737bea82f74383d6ed8f9fb8a9e844e5f716aea9f6b90d0: Status 404 returned error can't find the container with id ed1ac5e4aa8f59cc6737bea82f74383d6ed8f9fb8a9e844e5f716aea9f6b90d0
Oct 30 20:39:10.890751 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:10.890712    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5bww9" event={"ID":"6fdde40e-3330-4b84-a470-6a7e6a536dce","Type":"ContainerStarted","Data":"ed1ac5e4aa8f59cc6737bea82f74383d6ed8f9fb8a9e844e5f716aea9f6b90d0"}
Oct 30 20:39:11.896630 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:11.896594    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5bww9" event={"ID":"6fdde40e-3330-4b84-a470-6a7e6a536dce","Type":"ContainerStarted","Data":"38b603c06042b8200c7dd90dd4e22dc24036cbfc891fe895b6e5008e87a1e358"}
Oct 30 20:39:11.897015 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:11.896677    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5bww9" event={"ID":"6fdde40e-3330-4b84-a470-6a7e6a536dce","Type":"ContainerStarted","Data":"bfc7fe3122d974a796f0befe0ccaf8f9a708e84c78da936de4a4675c5b122a22"}
Oct 30 20:39:11.948577 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:11.948544    4521 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-9tfw6"]
Oct 30 20:39:11.951785 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:11.951762    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-9tfw6"
Oct 30 20:39:11.959947 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:11.959592    4521 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Oct 30 20:39:11.960101 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:11.960081    4521 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-h8wcz\""
Oct 30 20:39:11.960558 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:11.960534    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-9tfw6"]
Oct 30 20:39:12.014156 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:12.014118    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/f7db4cd9-6dc6-4201-9c54-3c52be1c889e-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-9tfw6\" (UID: \"f7db4cd9-6dc6-4201-9c54-3c52be1c889e\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-9tfw6"
Oct 30 20:39:12.115524 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:12.115466    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/f7db4cd9-6dc6-4201-9c54-3c52be1c889e-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-9tfw6\" (UID: \"f7db4cd9-6dc6-4201-9c54-3c52be1c889e\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-9tfw6"
Oct 30 20:39:12.117895 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:12.117867    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/f7db4cd9-6dc6-4201-9c54-3c52be1c889e-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-9tfw6\" (UID: \"f7db4cd9-6dc6-4201-9c54-3c52be1c889e\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-9tfw6"
Oct 30 20:39:12.263470 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:12.263430    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-9tfw6"
Oct 30 20:39:12.533517 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:12.533437    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-9tfw6"]
Oct 30 20:39:12.537038 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:39:12.537012    4521 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7db4cd9_6dc6_4201_9c54_3c52be1c889e.slice/crio-647fb1a0a42695d5792e7ca7c4d30a0d6d891f8c268510c32ed2cdaa28643d2a WatchSource:0}: Error finding container 647fb1a0a42695d5792e7ca7c4d30a0d6d891f8c268510c32ed2cdaa28643d2a: Status 404 returned error can't find the container with id 647fb1a0a42695d5792e7ca7c4d30a0d6d891f8c268510c32ed2cdaa28643d2a
Oct 30 20:39:12.900882 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:12.900798    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-9tfw6" event={"ID":"f7db4cd9-6dc6-4201-9c54-3c52be1c889e","Type":"ContainerStarted","Data":"647fb1a0a42695d5792e7ca7c4d30a0d6d891f8c268510c32ed2cdaa28643d2a"}
Oct 30 20:39:13.908599 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:13.908506    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-9tfw6" event={"ID":"f7db4cd9-6dc6-4201-9c54-3c52be1c889e","Type":"ContainerStarted","Data":"391a2feae2febcab3395302f9c36a8c48812b8851d47aecccf5055375eaf1548"}
Oct 30 20:39:13.909025 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:13.908691    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-9tfw6"
Oct 30 20:39:13.913383 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:13.913360    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-9tfw6"
Oct 30 20:39:13.927002 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:13.926946    4521 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-9tfw6" podStartSLOduration=1.8533444239999999 podStartE2EDuration="2.926931465s" totalImagesPullingTime="1.073587041s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:39:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:39:13.9267256 +0000 UTC m=+21.272346839" watchObservedRunningTime="2026-10-30 20:39:13.926931465 +0000 UTC m=+21.272552696"
Oct 30 20:39:16.427348 ip-10-0-131-125 kubenswrapper[4521]: E1030 20:39:16.427294    4521 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lkn2k" containerName="extractor"
Oct 30 20:39:20.531473 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:20.531442    4521 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-5s5hx"]
Oct 30 20:39:20.536552 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:20.536527    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-5s5hx"
Oct 30 20:39:20.553182 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:20.552866    4521 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 20:39:20.553182 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:20.552904    4521 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 30 20:39:20.553393 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:20.553319    4521 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 30 20:39:20.553891 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:20.553865    4521 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 30 20:39:20.576562 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:20.576527    4521 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 30 20:39:20.577403 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:20.577381    4521 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 30 20:39:20.577542 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:20.577443    4521 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-rmftm\""
Oct 30 20:39:20.671388 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:20.671350    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lc9wl\" (UniqueName: \"kubernetes.io/projected/df915ea2-605d-413e-8942-083da7b4ba8f-kube-api-access-lc9wl\") pod \"node-exporter-5s5hx\" (UID: \"df915ea2-605d-413e-8942-083da7b4ba8f\") " pod="openshift-monitoring/node-exporter-5s5hx"
Oct 30 20:39:20.671388 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:20.671391    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/df915ea2-605d-413e-8942-083da7b4ba8f-node-exporter-accelerators-collector-config\") pod \"node-exporter-5s5hx\" (UID: \"df915ea2-605d-413e-8942-083da7b4ba8f\") " pod="openshift-monitoring/node-exporter-5s5hx"
Oct 30 20:39:20.671644 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:20.671410    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/df915ea2-605d-413e-8942-083da7b4ba8f-root\") pod \"node-exporter-5s5hx\" (UID: \"df915ea2-605d-413e-8942-083da7b4ba8f\") " pod="openshift-monitoring/node-exporter-5s5hx"
Oct 30 20:39:20.671644 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:20.671428    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/df915ea2-605d-413e-8942-083da7b4ba8f-sys\") pod \"node-exporter-5s5hx\" (UID: \"df915ea2-605d-413e-8942-083da7b4ba8f\") " pod="openshift-monitoring/node-exporter-5s5hx"
Oct 30 20:39:20.671644 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:20.671601    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/df915ea2-605d-413e-8942-083da7b4ba8f-node-exporter-textfile\") pod \"node-exporter-5s5hx\" (UID: \"df915ea2-605d-413e-8942-083da7b4ba8f\") " pod="openshift-monitoring/node-exporter-5s5hx"
Oct 30 20:39:20.671644 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:20.671630    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/df915ea2-605d-413e-8942-083da7b4ba8f-node-exporter-wtmp\") pod \"node-exporter-5s5hx\" (UID: \"df915ea2-605d-413e-8942-083da7b4ba8f\") " pod="openshift-monitoring/node-exporter-5s5hx"
Oct 30 20:39:20.671836 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:20.671729    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/df915ea2-605d-413e-8942-083da7b4ba8f-node-exporter-tls\") pod \"node-exporter-5s5hx\" (UID: \"df915ea2-605d-413e-8942-083da7b4ba8f\") " pod="openshift-monitoring/node-exporter-5s5hx"
Oct 30 20:39:20.671836 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:20.671770    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/df915ea2-605d-413e-8942-083da7b4ba8f-metrics-client-ca\") pod \"node-exporter-5s5hx\" (UID: \"df915ea2-605d-413e-8942-083da7b4ba8f\") " pod="openshift-monitoring/node-exporter-5s5hx"
Oct 30 20:39:20.671910 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:20.671835    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/df915ea2-605d-413e-8942-083da7b4ba8f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5s5hx\" (UID: \"df915ea2-605d-413e-8942-083da7b4ba8f\") " pod="openshift-monitoring/node-exporter-5s5hx"
Oct 30 20:39:20.772402 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:20.772362    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/df915ea2-605d-413e-8942-083da7b4ba8f-node-exporter-textfile\") pod \"node-exporter-5s5hx\" (UID: \"df915ea2-605d-413e-8942-083da7b4ba8f\") " pod="openshift-monitoring/node-exporter-5s5hx"
Oct 30 20:39:20.772595 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:20.772415    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/df915ea2-605d-413e-8942-083da7b4ba8f-node-exporter-wtmp\") pod \"node-exporter-5s5hx\" (UID: \"df915ea2-605d-413e-8942-083da7b4ba8f\") " pod="openshift-monitoring/node-exporter-5s5hx"
Oct 30 20:39:20.772595 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:20.772443    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/df915ea2-605d-413e-8942-083da7b4ba8f-node-exporter-tls\") pod \"node-exporter-5s5hx\" (UID: \"df915ea2-605d-413e-8942-083da7b4ba8f\") " pod="openshift-monitoring/node-exporter-5s5hx"
Oct 30 20:39:20.772595 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:20.772465    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/df915ea2-605d-413e-8942-083da7b4ba8f-metrics-client-ca\") pod \"node-exporter-5s5hx\" (UID: \"df915ea2-605d-413e-8942-083da7b4ba8f\") " pod="openshift-monitoring/node-exporter-5s5hx"
Oct 30 20:39:20.772595 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:20.772488    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/df915ea2-605d-413e-8942-083da7b4ba8f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5s5hx\" (UID: \"df915ea2-605d-413e-8942-083da7b4ba8f\") " pod="openshift-monitoring/node-exporter-5s5hx"
Oct 30 20:39:20.772595 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:20.772522    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lc9wl\" (UniqueName: \"kubernetes.io/projected/df915ea2-605d-413e-8942-083da7b4ba8f-kube-api-access-lc9wl\") pod \"node-exporter-5s5hx\" (UID: \"df915ea2-605d-413e-8942-083da7b4ba8f\") " pod="openshift-monitoring/node-exporter-5s5hx"
Oct 30 20:39:20.772595 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:20.772552    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/df915ea2-605d-413e-8942-083da7b4ba8f-node-exporter-accelerators-collector-config\") pod \"node-exporter-5s5hx\" (UID: \"df915ea2-605d-413e-8942-083da7b4ba8f\") " pod="openshift-monitoring/node-exporter-5s5hx"
Oct 30 20:39:20.772595 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:20.772577    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/df915ea2-605d-413e-8942-083da7b4ba8f-root\") pod \"node-exporter-5s5hx\" (UID: \"df915ea2-605d-413e-8942-083da7b4ba8f\") " pod="openshift-monitoring/node-exporter-5s5hx"
Oct 30 20:39:20.772934 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:20.772604    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/df915ea2-605d-413e-8942-083da7b4ba8f-sys\") pod \"node-exporter-5s5hx\" (UID: \"df915ea2-605d-413e-8942-083da7b4ba8f\") " pod="openshift-monitoring/node-exporter-5s5hx"
Oct 30 20:39:20.772934 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:20.772680    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/df915ea2-605d-413e-8942-083da7b4ba8f-node-exporter-wtmp\") pod \"node-exporter-5s5hx\" (UID: \"df915ea2-605d-413e-8942-083da7b4ba8f\") " pod="openshift-monitoring/node-exporter-5s5hx"
Oct 30 20:39:20.772934 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:20.772688    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/df915ea2-605d-413e-8942-083da7b4ba8f-sys\") pod \"node-exporter-5s5hx\" (UID: \"df915ea2-605d-413e-8942-083da7b4ba8f\") " pod="openshift-monitoring/node-exporter-5s5hx"
Oct 30 20:39:20.772934 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:20.772780    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/df915ea2-605d-413e-8942-083da7b4ba8f-node-exporter-textfile\") pod \"node-exporter-5s5hx\" (UID: \"df915ea2-605d-413e-8942-083da7b4ba8f\") " pod="openshift-monitoring/node-exporter-5s5hx"
Oct 30 20:39:20.772934 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:20.772852    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/df915ea2-605d-413e-8942-083da7b4ba8f-root\") pod \"node-exporter-5s5hx\" (UID: \"df915ea2-605d-413e-8942-083da7b4ba8f\") " pod="openshift-monitoring/node-exporter-5s5hx"
Oct 30 20:39:20.773187 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:20.773169    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/df915ea2-605d-413e-8942-083da7b4ba8f-node-exporter-accelerators-collector-config\") pod \"node-exporter-5s5hx\" (UID: \"df915ea2-605d-413e-8942-083da7b4ba8f\") " pod="openshift-monitoring/node-exporter-5s5hx"
Oct 30 20:39:20.773235 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:20.773166    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/df915ea2-605d-413e-8942-083da7b4ba8f-metrics-client-ca\") pod \"node-exporter-5s5hx\" (UID: \"df915ea2-605d-413e-8942-083da7b4ba8f\") " pod="openshift-monitoring/node-exporter-5s5hx"
Oct 30 20:39:20.774897 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:20.774877    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/df915ea2-605d-413e-8942-083da7b4ba8f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5s5hx\" (UID: \"df915ea2-605d-413e-8942-083da7b4ba8f\") " pod="openshift-monitoring/node-exporter-5s5hx"
Oct 30 20:39:20.774968 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:20.774952    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/df915ea2-605d-413e-8942-083da7b4ba8f-node-exporter-tls\") pod \"node-exporter-5s5hx\" (UID: \"df915ea2-605d-413e-8942-083da7b4ba8f\") " pod="openshift-monitoring/node-exporter-5s5hx"
Oct 30 20:39:20.789623 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:20.789563    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lc9wl\" (UniqueName: \"kubernetes.io/projected/df915ea2-605d-413e-8942-083da7b4ba8f-kube-api-access-lc9wl\") pod \"node-exporter-5s5hx\" (UID: \"df915ea2-605d-413e-8942-083da7b4ba8f\") " pod="openshift-monitoring/node-exporter-5s5hx"
Oct 30 20:39:20.853535 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:20.853500    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-5s5hx"
Oct 30 20:39:20.864652 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:39:20.864617    4521 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf915ea2_605d_413e_8942_083da7b4ba8f.slice/crio-9dc68776cc9b681ae90eeb1ffc534e61aeccbc23adccf1857b1a46742ab45f00 WatchSource:0}: Error finding container 9dc68776cc9b681ae90eeb1ffc534e61aeccbc23adccf1857b1a46742ab45f00: Status 404 returned error can't find the container with id 9dc68776cc9b681ae90eeb1ffc534e61aeccbc23adccf1857b1a46742ab45f00
Oct 30 20:39:20.934658 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:20.934627    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5s5hx" event={"ID":"df915ea2-605d-413e-8942-083da7b4ba8f","Type":"ContainerStarted","Data":"9dc68776cc9b681ae90eeb1ffc534e61aeccbc23adccf1857b1a46742ab45f00"}
Oct 30 20:39:22.942677 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:22.942635    4521 generic.go:356] "Generic (PLEG): container finished" podID="df915ea2-605d-413e-8942-083da7b4ba8f" containerID="288ecd2338fbc994c6f4cb244997530d4acd2a7423f3d63ef33cdbccdd2198b1" exitCode=0
Oct 30 20:39:22.943068 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:22.942719    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5s5hx" event={"ID":"df915ea2-605d-413e-8942-083da7b4ba8f","Type":"ContainerDied","Data":"288ecd2338fbc994c6f4cb244997530d4acd2a7423f3d63ef33cdbccdd2198b1"}
Oct 30 20:39:23.948397 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:23.948358    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5s5hx" event={"ID":"df915ea2-605d-413e-8942-083da7b4ba8f","Type":"ContainerStarted","Data":"3b6bec9de967eb283f9e5d76ef6a3c02e2840d94ead1adc1dcd3584868fe4403"}
Oct 30 20:39:23.948846 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:23.948442    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5s5hx" event={"ID":"df915ea2-605d-413e-8942-083da7b4ba8f","Type":"ContainerStarted","Data":"80d558ab7e786883fe0360a76706391c9456d23e93af868ae1f44edbf5b5eef0"}
Oct 30 20:39:23.971912 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:23.971871    4521 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-5s5hx" podStartSLOduration=2.858231185 podStartE2EDuration="3.971859466s" totalImagesPullingTime="1.044647772s" totalInitContainerRuntime="68.980509ms" isStatefulPod=true podCreationTimestamp="2026-10-30 20:39:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:39:23.971134163 +0000 UTC m=+31.316755396" watchObservedRunningTime="2026-10-30 20:39:23.971859466 +0000 UTC m=+31.317480731"
Oct 30 20:39:24.829707 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:24.829677    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-f4cch"
Oct 30 20:39:29.181716 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:29.181683    4521 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-ms68j"]
Oct 30 20:39:29.183996 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:29.183977    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-ms68j"
Oct 30 20:39:29.187982 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:29.187957    4521 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Oct 30 20:39:29.188082 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:29.188014    4521 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Oct 30 20:39:29.189078 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:29.189062    4521 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-52vcr\""
Oct 30 20:39:29.202613 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:29.202591    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-ms68j"]
Oct 30 20:39:29.332042 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:29.332005    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/426b83f8-572e-4178-adb4-bf1c2a6bee2b-tmp\") pod \"downloads-6dd789c8b7-ms68j\" (UID: \"426b83f8-572e-4178-adb4-bf1c2a6bee2b\") " pod="openshift-console/downloads-6dd789c8b7-ms68j"
Oct 30 20:39:29.332042 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:29.332055    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcltm\" (UniqueName: \"kubernetes.io/projected/426b83f8-572e-4178-adb4-bf1c2a6bee2b-kube-api-access-dcltm\") pod \"downloads-6dd789c8b7-ms68j\" (UID: \"426b83f8-572e-4178-adb4-bf1c2a6bee2b\") " pod="openshift-console/downloads-6dd789c8b7-ms68j"
Oct 30 20:39:29.433233 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:29.433148    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/426b83f8-572e-4178-adb4-bf1c2a6bee2b-tmp\") pod \"downloads-6dd789c8b7-ms68j\" (UID: \"426b83f8-572e-4178-adb4-bf1c2a6bee2b\") " pod="openshift-console/downloads-6dd789c8b7-ms68j"
Oct 30 20:39:29.433233 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:29.433192    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcltm\" (UniqueName: \"kubernetes.io/projected/426b83f8-572e-4178-adb4-bf1c2a6bee2b-kube-api-access-dcltm\") pod \"downloads-6dd789c8b7-ms68j\" (UID: \"426b83f8-572e-4178-adb4-bf1c2a6bee2b\") " pod="openshift-console/downloads-6dd789c8b7-ms68j"
Oct 30 20:39:29.433575 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:29.433556    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/426b83f8-572e-4178-adb4-bf1c2a6bee2b-tmp\") pod \"downloads-6dd789c8b7-ms68j\" (UID: \"426b83f8-572e-4178-adb4-bf1c2a6bee2b\") " pod="openshift-console/downloads-6dd789c8b7-ms68j"
Oct 30 20:39:29.445974 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:29.445942    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcltm\" (UniqueName: \"kubernetes.io/projected/426b83f8-572e-4178-adb4-bf1c2a6bee2b-kube-api-access-dcltm\") pod \"downloads-6dd789c8b7-ms68j\" (UID: \"426b83f8-572e-4178-adb4-bf1c2a6bee2b\") " pod="openshift-console/downloads-6dd789c8b7-ms68j"
Oct 30 20:39:29.493527 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:29.493493    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-ms68j"
Oct 30 20:39:29.765953 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:29.765918    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-ms68j"]
Oct 30 20:39:29.769835 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:39:29.769804    4521 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod426b83f8_572e_4178_adb4_bf1c2a6bee2b.slice/crio-84f21c9261e65a4b2f4f14067c32ec5799c32dcb363437e5dd192ab6728cfb3b WatchSource:0}: Error finding container 84f21c9261e65a4b2f4f14067c32ec5799c32dcb363437e5dd192ab6728cfb3b: Status 404 returned error can't find the container with id 84f21c9261e65a4b2f4f14067c32ec5799c32dcb363437e5dd192ab6728cfb3b
Oct 30 20:39:29.968083 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:29.968047    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-ms68j" event={"ID":"426b83f8-572e-4178-adb4-bf1c2a6bee2b","Type":"ContainerStarted","Data":"84f21c9261e65a4b2f4f14067c32ec5799c32dcb363437e5dd192ab6728cfb3b"}
Oct 30 20:39:30.896910 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:30.896155    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7579767596-gmrh5"
Oct 30 20:39:38.776443 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:38.776404    4521 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-74cc97464-lhsft"]
Oct 30 20:39:38.779596 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:38.779559    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74cc97464-lhsft"
Oct 30 20:39:38.789683 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:38.789658    4521 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Oct 30 20:39:38.789683 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:38.789663    4521 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Oct 30 20:39:38.789874 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:38.789690    4521 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Oct 30 20:39:38.789874 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:38.789697    4521 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Oct 30 20:39:38.789874 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:38.789741    4521 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-492z7\""
Oct 30 20:39:38.789874 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:38.789784    4521 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Oct 30 20:39:38.792986 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:38.792964    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-74cc97464-lhsft"]
Oct 30 20:39:38.909275 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:38.909237    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e3915ee4-ed6d-43e1-816c-b33e05bf78e0-console-config\") pod \"console-74cc97464-lhsft\" (UID: \"e3915ee4-ed6d-43e1-816c-b33e05bf78e0\") " pod="openshift-console/console-74cc97464-lhsft"
Oct 30 20:39:38.909482 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:38.909304    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e3915ee4-ed6d-43e1-816c-b33e05bf78e0-console-oauth-config\") pod \"console-74cc97464-lhsft\" (UID: \"e3915ee4-ed6d-43e1-816c-b33e05bf78e0\") " pod="openshift-console/console-74cc97464-lhsft"
Oct 30 20:39:38.909482 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:38.909419    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e3915ee4-ed6d-43e1-816c-b33e05bf78e0-oauth-serving-cert\") pod \"console-74cc97464-lhsft\" (UID: \"e3915ee4-ed6d-43e1-816c-b33e05bf78e0\") " pod="openshift-console/console-74cc97464-lhsft"
Oct 30 20:39:38.909482 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:38.909452    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e3915ee4-ed6d-43e1-816c-b33e05bf78e0-console-serving-cert\") pod \"console-74cc97464-lhsft\" (UID: \"e3915ee4-ed6d-43e1-816c-b33e05bf78e0\") " pod="openshift-console/console-74cc97464-lhsft"
Oct 30 20:39:38.909482 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:38.909477    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e3915ee4-ed6d-43e1-816c-b33e05bf78e0-service-ca\") pod \"console-74cc97464-lhsft\" (UID: \"e3915ee4-ed6d-43e1-816c-b33e05bf78e0\") " pod="openshift-console/console-74cc97464-lhsft"
Oct 30 20:39:38.909650 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:38.909554    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nmv2\" (UniqueName: \"kubernetes.io/projected/e3915ee4-ed6d-43e1-816c-b33e05bf78e0-kube-api-access-9nmv2\") pod \"console-74cc97464-lhsft\" (UID: \"e3915ee4-ed6d-43e1-816c-b33e05bf78e0\") " pod="openshift-console/console-74cc97464-lhsft"
Oct 30 20:39:38.909682 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:38.909642    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e3915ee4-ed6d-43e1-816c-b33e05bf78e0-tmp\") pod \"console-74cc97464-lhsft\" (UID: \"e3915ee4-ed6d-43e1-816c-b33e05bf78e0\") " pod="openshift-console/console-74cc97464-lhsft"
Oct 30 20:39:39.010737 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:39.010692    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nmv2\" (UniqueName: \"kubernetes.io/projected/e3915ee4-ed6d-43e1-816c-b33e05bf78e0-kube-api-access-9nmv2\") pod \"console-74cc97464-lhsft\" (UID: \"e3915ee4-ed6d-43e1-816c-b33e05bf78e0\") " pod="openshift-console/console-74cc97464-lhsft"
Oct 30 20:39:39.010920 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:39.010744    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e3915ee4-ed6d-43e1-816c-b33e05bf78e0-tmp\") pod \"console-74cc97464-lhsft\" (UID: \"e3915ee4-ed6d-43e1-816c-b33e05bf78e0\") " pod="openshift-console/console-74cc97464-lhsft"
Oct 30 20:39:39.010920 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:39.010796    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e3915ee4-ed6d-43e1-816c-b33e05bf78e0-console-config\") pod \"console-74cc97464-lhsft\" (UID: \"e3915ee4-ed6d-43e1-816c-b33e05bf78e0\") " pod="openshift-console/console-74cc97464-lhsft"
Oct 30 20:39:39.010920 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:39.010841    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e3915ee4-ed6d-43e1-816c-b33e05bf78e0-console-oauth-config\") pod \"console-74cc97464-lhsft\" (UID: \"e3915ee4-ed6d-43e1-816c-b33e05bf78e0\") " pod="openshift-console/console-74cc97464-lhsft"
Oct 30 20:39:39.010920 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:39.010867    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e3915ee4-ed6d-43e1-816c-b33e05bf78e0-oauth-serving-cert\") pod \"console-74cc97464-lhsft\" (UID: \"e3915ee4-ed6d-43e1-816c-b33e05bf78e0\") " pod="openshift-console/console-74cc97464-lhsft"
Oct 30 20:39:39.010920 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:39.010892    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e3915ee4-ed6d-43e1-816c-b33e05bf78e0-console-serving-cert\") pod \"console-74cc97464-lhsft\" (UID: \"e3915ee4-ed6d-43e1-816c-b33e05bf78e0\") " pod="openshift-console/console-74cc97464-lhsft"
Oct 30 20:39:39.010920 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:39.010918    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e3915ee4-ed6d-43e1-816c-b33e05bf78e0-service-ca\") pod \"console-74cc97464-lhsft\" (UID: \"e3915ee4-ed6d-43e1-816c-b33e05bf78e0\") " pod="openshift-console/console-74cc97464-lhsft"
Oct 30 20:39:39.011699 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:39.011672    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e3915ee4-ed6d-43e1-816c-b33e05bf78e0-tmp\") pod \"console-74cc97464-lhsft\" (UID: \"e3915ee4-ed6d-43e1-816c-b33e05bf78e0\") " pod="openshift-console/console-74cc97464-lhsft"
Oct 30 20:39:39.011812 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:39.011728    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e3915ee4-ed6d-43e1-816c-b33e05bf78e0-service-ca\") pod \"console-74cc97464-lhsft\" (UID: \"e3915ee4-ed6d-43e1-816c-b33e05bf78e0\") " pod="openshift-console/console-74cc97464-lhsft"
Oct 30 20:39:39.012005 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:39.011982    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e3915ee4-ed6d-43e1-816c-b33e05bf78e0-console-config\") pod \"console-74cc97464-lhsft\" (UID: \"e3915ee4-ed6d-43e1-816c-b33e05bf78e0\") " pod="openshift-console/console-74cc97464-lhsft"
Oct 30 20:39:39.012202 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:39.012181    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e3915ee4-ed6d-43e1-816c-b33e05bf78e0-oauth-serving-cert\") pod \"console-74cc97464-lhsft\" (UID: \"e3915ee4-ed6d-43e1-816c-b33e05bf78e0\") " pod="openshift-console/console-74cc97464-lhsft"
Oct 30 20:39:39.013675 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:39.013654    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e3915ee4-ed6d-43e1-816c-b33e05bf78e0-console-oauth-config\") pod \"console-74cc97464-lhsft\" (UID: \"e3915ee4-ed6d-43e1-816c-b33e05bf78e0\") " pod="openshift-console/console-74cc97464-lhsft"
Oct 30 20:39:39.014026 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:39.014004    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e3915ee4-ed6d-43e1-816c-b33e05bf78e0-console-serving-cert\") pod \"console-74cc97464-lhsft\" (UID: \"e3915ee4-ed6d-43e1-816c-b33e05bf78e0\") " pod="openshift-console/console-74cc97464-lhsft"
Oct 30 20:39:39.023655 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:39.023634    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nmv2\" (UniqueName: \"kubernetes.io/projected/e3915ee4-ed6d-43e1-816c-b33e05bf78e0-kube-api-access-9nmv2\") pod \"console-74cc97464-lhsft\" (UID: \"e3915ee4-ed6d-43e1-816c-b33e05bf78e0\") " pod="openshift-console/console-74cc97464-lhsft"
Oct 30 20:39:39.093152 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:39.093065    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74cc97464-lhsft"
Oct 30 20:39:39.383951 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:39.383826    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-74cc97464-lhsft"]
Oct 30 20:39:39.387367 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:39:39.387309    4521 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3915ee4_ed6d_43e1_816c_b33e05bf78e0.slice/crio-b853daa581d07a7cb29356e1c07e57dbdd698575b23d627be749b5fbb78edfa6 WatchSource:0}: Error finding container b853daa581d07a7cb29356e1c07e57dbdd698575b23d627be749b5fbb78edfa6: Status 404 returned error can't find the container with id b853daa581d07a7cb29356e1c07e57dbdd698575b23d627be749b5fbb78edfa6
Oct 30 20:39:40.005426 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:40.005372    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74cc97464-lhsft" event={"ID":"e3915ee4-ed6d-43e1-816c-b33e05bf78e0","Type":"ContainerStarted","Data":"b853daa581d07a7cb29356e1c07e57dbdd698575b23d627be749b5fbb78edfa6"}
Oct 30 20:39:43.018042 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:43.018008    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74cc97464-lhsft" event={"ID":"e3915ee4-ed6d-43e1-816c-b33e05bf78e0","Type":"ContainerStarted","Data":"9da7676dc8fd57ccd1c45183a3874f8b9c5ab699aa86c0ddd05fdc581287e1a4"}
Oct 30 20:39:43.040395 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:43.040300    4521 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-74cc97464-lhsft" podStartSLOduration=2.171151401 podStartE2EDuration="5.04028706s" totalImagesPullingTime="2.869135659s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:39:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:39:43.038943075 +0000 UTC m=+50.384564313" watchObservedRunningTime="2026-10-30 20:39:43.04028706 +0000 UTC m=+50.385908292"
Oct 30 20:39:46.031992 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:46.031932    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-ms68j" event={"ID":"426b83f8-572e-4178-adb4-bf1c2a6bee2b","Type":"ContainerStarted","Data":"5426a11b8f64301908ce3cc4ac185676d5e3e2d8112f75a1548fa2e562db2633"}
Oct 30 20:39:46.032476 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:46.032305    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-ms68j"
Oct 30 20:39:46.047965 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:46.047938    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-ms68j"
Oct 30 20:39:46.059939 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:46.059882    4521 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-ms68j" podStartSLOduration=1.53262187 podStartE2EDuration="17.059863982s" totalImagesPullingTime="15.527242112s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:39:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:39:46.058822634 +0000 UTC m=+53.404443884" watchObservedRunningTime="2026-10-30 20:39:46.059863982 +0000 UTC m=+53.405485221"
Oct 30 20:39:46.433107 ip-10-0-131-125 kubenswrapper[4521]: E1030 20:39:46.433012    4521 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lkn2k" containerName="extractor"
Oct 30 20:39:48.183360 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:48.183306    4521 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-8d77f6bfd-hr84v"]
Oct 30 20:39:48.209319 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:48.209289    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8d77f6bfd-hr84v"]
Oct 30 20:39:48.209492 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:48.209406    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8d77f6bfd-hr84v"
Oct 30 20:39:48.220998 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:48.220970    4521 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Oct 30 20:39:48.389951 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:48.389907    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkr29\" (UniqueName: \"kubernetes.io/projected/bf502e09-d489-4574-83e9-98232c580449-kube-api-access-nkr29\") pod \"console-8d77f6bfd-hr84v\" (UID: \"bf502e09-d489-4574-83e9-98232c580449\") " pod="openshift-console/console-8d77f6bfd-hr84v"
Oct 30 20:39:48.390121 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:48.389966    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bf502e09-d489-4574-83e9-98232c580449-console-oauth-config\") pod \"console-8d77f6bfd-hr84v\" (UID: \"bf502e09-d489-4574-83e9-98232c580449\") " pod="openshift-console/console-8d77f6bfd-hr84v"
Oct 30 20:39:48.390121 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:48.390083    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bf502e09-d489-4574-83e9-98232c580449-console-serving-cert\") pod \"console-8d77f6bfd-hr84v\" (UID: \"bf502e09-d489-4574-83e9-98232c580449\") " pod="openshift-console/console-8d77f6bfd-hr84v"
Oct 30 20:39:48.390244 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:48.390127    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bf502e09-d489-4574-83e9-98232c580449-trusted-ca-bundle\") pod \"console-8d77f6bfd-hr84v\" (UID: \"bf502e09-d489-4574-83e9-98232c580449\") " pod="openshift-console/console-8d77f6bfd-hr84v"
Oct 30 20:39:48.390244 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:48.390156    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bf502e09-d489-4574-83e9-98232c580449-tmp\") pod \"console-8d77f6bfd-hr84v\" (UID: \"bf502e09-d489-4574-83e9-98232c580449\") " pod="openshift-console/console-8d77f6bfd-hr84v"
Oct 30 20:39:48.390244 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:48.390177    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bf502e09-d489-4574-83e9-98232c580449-service-ca\") pod \"console-8d77f6bfd-hr84v\" (UID: \"bf502e09-d489-4574-83e9-98232c580449\") " pod="openshift-console/console-8d77f6bfd-hr84v"
Oct 30 20:39:48.390410 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:48.390278    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bf502e09-d489-4574-83e9-98232c580449-oauth-serving-cert\") pod \"console-8d77f6bfd-hr84v\" (UID: \"bf502e09-d489-4574-83e9-98232c580449\") " pod="openshift-console/console-8d77f6bfd-hr84v"
Oct 30 20:39:48.390410 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:48.390319    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bf502e09-d489-4574-83e9-98232c580449-console-config\") pod \"console-8d77f6bfd-hr84v\" (UID: \"bf502e09-d489-4574-83e9-98232c580449\") " pod="openshift-console/console-8d77f6bfd-hr84v"
Oct 30 20:39:48.491090 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:48.491059    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bf502e09-d489-4574-83e9-98232c580449-console-oauth-config\") pod \"console-8d77f6bfd-hr84v\" (UID: \"bf502e09-d489-4574-83e9-98232c580449\") " pod="openshift-console/console-8d77f6bfd-hr84v"
Oct 30 20:39:48.491269 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:48.491100    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bf502e09-d489-4574-83e9-98232c580449-console-serving-cert\") pod \"console-8d77f6bfd-hr84v\" (UID: \"bf502e09-d489-4574-83e9-98232c580449\") " pod="openshift-console/console-8d77f6bfd-hr84v"
Oct 30 20:39:48.491269 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:48.491244    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bf502e09-d489-4574-83e9-98232c580449-trusted-ca-bundle\") pod \"console-8d77f6bfd-hr84v\" (UID: \"bf502e09-d489-4574-83e9-98232c580449\") " pod="openshift-console/console-8d77f6bfd-hr84v"
Oct 30 20:39:48.491458 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:48.491294    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bf502e09-d489-4574-83e9-98232c580449-tmp\") pod \"console-8d77f6bfd-hr84v\" (UID: \"bf502e09-d489-4574-83e9-98232c580449\") " pod="openshift-console/console-8d77f6bfd-hr84v"
Oct 30 20:39:48.491458 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:48.491340    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bf502e09-d489-4574-83e9-98232c580449-service-ca\") pod \"console-8d77f6bfd-hr84v\" (UID: \"bf502e09-d489-4574-83e9-98232c580449\") " pod="openshift-console/console-8d77f6bfd-hr84v"
Oct 30 20:39:48.491458 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:48.491404    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bf502e09-d489-4574-83e9-98232c580449-oauth-serving-cert\") pod \"console-8d77f6bfd-hr84v\" (UID: \"bf502e09-d489-4574-83e9-98232c580449\") " pod="openshift-console/console-8d77f6bfd-hr84v"
Oct 30 20:39:48.491458 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:48.491440    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bf502e09-d489-4574-83e9-98232c580449-console-config\") pod \"console-8d77f6bfd-hr84v\" (UID: \"bf502e09-d489-4574-83e9-98232c580449\") " pod="openshift-console/console-8d77f6bfd-hr84v"
Oct 30 20:39:48.491673 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:48.491478    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkr29\" (UniqueName: \"kubernetes.io/projected/bf502e09-d489-4574-83e9-98232c580449-kube-api-access-nkr29\") pod \"console-8d77f6bfd-hr84v\" (UID: \"bf502e09-d489-4574-83e9-98232c580449\") " pod="openshift-console/console-8d77f6bfd-hr84v"
Oct 30 20:39:48.491729 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:48.491706    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bf502e09-d489-4574-83e9-98232c580449-tmp\") pod \"console-8d77f6bfd-hr84v\" (UID: \"bf502e09-d489-4574-83e9-98232c580449\") " pod="openshift-console/console-8d77f6bfd-hr84v"
Oct 30 20:39:48.492528 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:48.492178    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bf502e09-d489-4574-83e9-98232c580449-oauth-serving-cert\") pod \"console-8d77f6bfd-hr84v\" (UID: \"bf502e09-d489-4574-83e9-98232c580449\") " pod="openshift-console/console-8d77f6bfd-hr84v"
Oct 30 20:39:48.492528 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:48.492360    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bf502e09-d489-4574-83e9-98232c580449-service-ca\") pod \"console-8d77f6bfd-hr84v\" (UID: \"bf502e09-d489-4574-83e9-98232c580449\") " pod="openshift-console/console-8d77f6bfd-hr84v"
Oct 30 20:39:48.492528 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:48.492475    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bf502e09-d489-4574-83e9-98232c580449-console-config\") pod \"console-8d77f6bfd-hr84v\" (UID: \"bf502e09-d489-4574-83e9-98232c580449\") " pod="openshift-console/console-8d77f6bfd-hr84v"
Oct 30 20:39:48.493032 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:48.492905    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bf502e09-d489-4574-83e9-98232c580449-trusted-ca-bundle\") pod \"console-8d77f6bfd-hr84v\" (UID: \"bf502e09-d489-4574-83e9-98232c580449\") " pod="openshift-console/console-8d77f6bfd-hr84v"
Oct 30 20:39:48.494068 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:48.494043    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bf502e09-d489-4574-83e9-98232c580449-console-oauth-config\") pod \"console-8d77f6bfd-hr84v\" (UID: \"bf502e09-d489-4574-83e9-98232c580449\") " pod="openshift-console/console-8d77f6bfd-hr84v"
Oct 30 20:39:48.494615 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:48.494594    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bf502e09-d489-4574-83e9-98232c580449-console-serving-cert\") pod \"console-8d77f6bfd-hr84v\" (UID: \"bf502e09-d489-4574-83e9-98232c580449\") " pod="openshift-console/console-8d77f6bfd-hr84v"
Oct 30 20:39:48.506953 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:48.506927    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkr29\" (UniqueName: \"kubernetes.io/projected/bf502e09-d489-4574-83e9-98232c580449-kube-api-access-nkr29\") pod \"console-8d77f6bfd-hr84v\" (UID: \"bf502e09-d489-4574-83e9-98232c580449\") " pod="openshift-console/console-8d77f6bfd-hr84v"
Oct 30 20:39:48.537238 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:48.537198    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8d77f6bfd-hr84v"
Oct 30 20:39:48.848634 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:48.848560    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8d77f6bfd-hr84v"]
Oct 30 20:39:48.855740 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:39:48.855703    4521 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf502e09_d489_4574_83e9_98232c580449.slice/crio-0ec23e92252dbb24c8500aa4109a84c610c30667421574e6e7ebacabfbf13028 WatchSource:0}: Error finding container 0ec23e92252dbb24c8500aa4109a84c610c30667421574e6e7ebacabfbf13028: Status 404 returned error can't find the container with id 0ec23e92252dbb24c8500aa4109a84c610c30667421574e6e7ebacabfbf13028
Oct 30 20:39:49.044964 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:49.044928    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8d77f6bfd-hr84v" event={"ID":"bf502e09-d489-4574-83e9-98232c580449","Type":"ContainerStarted","Data":"abf99511ab7e13234638e624526730678b8c3668b176ca51f38ebe144a87c989"}
Oct 30 20:39:49.045210 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:49.045180    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8d77f6bfd-hr84v" event={"ID":"bf502e09-d489-4574-83e9-98232c580449","Type":"ContainerStarted","Data":"0ec23e92252dbb24c8500aa4109a84c610c30667421574e6e7ebacabfbf13028"}
Oct 30 20:39:49.072511 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:49.072448    4521 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-8d77f6bfd-hr84v" podStartSLOduration=1.072433684 podStartE2EDuration="1.072433684s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:39:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:39:49.071914148 +0000 UTC m=+56.417535391" watchObservedRunningTime="2026-10-30 20:39:49.072433684 +0000 UTC m=+56.418054914"
Oct 30 20:39:49.094157 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:49.094126    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-74cc97464-lhsft"
Oct 30 20:39:49.094303 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:49.094180    4521 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-74cc97464-lhsft"
Oct 30 20:39:49.099663 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:49.099594    4521 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-74cc97464-lhsft"
Oct 30 20:39:50.053235 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:50.053205    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-74cc97464-lhsft"
Oct 30 20:39:58.538316 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:58.538279    4521 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-8d77f6bfd-hr84v"
Oct 30 20:39:58.538807 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:58.538388    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-8d77f6bfd-hr84v"
Oct 30 20:39:58.542946 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:58.542923    4521 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-8d77f6bfd-hr84v"
Oct 30 20:39:59.015775 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:59.015734    4521 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-4f444-dttkw"]
Oct 30 20:39:59.016174 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:59.016145    4521 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-4f444-dttkw" podUID="9033c4e2-4586-4ec9-996b-21c412cfb8ba" containerName="insights-gathering" containerID="cri-o://84f75d512bda86246800d98992feae6f917091e0b59369751f878ee85d0ae742" gracePeriod=30
Oct 30 20:39:59.080483 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:59.080455    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-8d77f6bfd-hr84v"
Oct 30 20:39:59.164553 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:59.164520    4521 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-74cc97464-lhsft"]
Oct 30 20:39:59.177349 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:59.176693    4521 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-stq6m-cr8z8"]
Oct 30 20:39:59.188128 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:59.188098    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-stq6m-cr8z8"
Oct 30 20:39:59.193571 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:59.193546    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-stq6m-cr8z8"]
Oct 30 20:39:59.278353 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:59.278245    4521 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-6cf8db96c7-kw294"]
Oct 30 20:39:59.279873 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:59.279847    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxgrg\" (UniqueName: \"kubernetes.io/projected/4bb3e3ca-18ce-4d77-9150-30ca3adfc11f-kube-api-access-dxgrg\") pod \"periodic-gathering-stq6m-cr8z8\" (UID: \"4bb3e3ca-18ce-4d77-9150-30ca3adfc11f\") " pod="openshift-insights/periodic-gathering-stq6m-cr8z8"
Oct 30 20:39:59.279974 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:59.279927    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4bb3e3ca-18ce-4d77-9150-30ca3adfc11f-trusted-ca-bundle\") pod \"periodic-gathering-stq6m-cr8z8\" (UID: \"4bb3e3ca-18ce-4d77-9150-30ca3adfc11f\") " pod="openshift-insights/periodic-gathering-stq6m-cr8z8"
Oct 30 20:39:59.280040 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:59.280017    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/4bb3e3ca-18ce-4d77-9150-30ca3adfc11f-archives-path\") pod \"periodic-gathering-stq6m-cr8z8\" (UID: \"4bb3e3ca-18ce-4d77-9150-30ca3adfc11f\") " pod="openshift-insights/periodic-gathering-stq6m-cr8z8"
Oct 30 20:39:59.280087 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:59.280071    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4bb3e3ca-18ce-4d77-9150-30ca3adfc11f-service-ca-bundle\") pod \"periodic-gathering-stq6m-cr8z8\" (UID: \"4bb3e3ca-18ce-4d77-9150-30ca3adfc11f\") " pod="openshift-insights/periodic-gathering-stq6m-cr8z8"
Oct 30 20:39:59.304438 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:59.304412    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6cf8db96c7-kw294"]
Oct 30 20:39:59.304589 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:59.304506    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6cf8db96c7-kw294"
Oct 30 20:39:59.312620 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:59.312598    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-4f444-dttkw"
Oct 30 20:39:59.380756 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:59.380721    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9033c4e2-4586-4ec9-996b-21c412cfb8ba-kube-api-access-2lvnm\" (UniqueName: \"kubernetes.io/projected/9033c4e2-4586-4ec9-996b-21c412cfb8ba-kube-api-access-2lvnm\") pod \"9033c4e2-4586-4ec9-996b-21c412cfb8ba\" (UID: \"9033c4e2-4586-4ec9-996b-21c412cfb8ba\") "
Oct 30 20:39:59.380756 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:59.380755    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9033c4e2-4586-4ec9-996b-21c412cfb8ba-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9033c4e2-4586-4ec9-996b-21c412cfb8ba-trusted-ca-bundle\") pod \"9033c4e2-4586-4ec9-996b-21c412cfb8ba\" (UID: \"9033c4e2-4586-4ec9-996b-21c412cfb8ba\") "
Oct 30 20:39:59.381011 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:59.380793    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9033c4e2-4586-4ec9-996b-21c412cfb8ba-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/9033c4e2-4586-4ec9-996b-21c412cfb8ba-archives-path\") pod \"9033c4e2-4586-4ec9-996b-21c412cfb8ba\" (UID: \"9033c4e2-4586-4ec9-996b-21c412cfb8ba\") "
Oct 30 20:39:59.381011 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:59.380826    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9033c4e2-4586-4ec9-996b-21c412cfb8ba-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9033c4e2-4586-4ec9-996b-21c412cfb8ba-service-ca-bundle\") pod \"9033c4e2-4586-4ec9-996b-21c412cfb8ba\" (UID: \"9033c4e2-4586-4ec9-996b-21c412cfb8ba\") "
Oct 30 20:39:59.381011 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:59.380903    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gqlh\" (UniqueName: \"kubernetes.io/projected/9241b135-3961-4ef5-9fcd-96a078e2475f-kube-api-access-9gqlh\") pod \"console-6cf8db96c7-kw294\" (UID: \"9241b135-3961-4ef5-9fcd-96a078e2475f\") " pod="openshift-console/console-6cf8db96c7-kw294"
Oct 30 20:39:59.381011 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:59.380938    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxgrg\" (UniqueName: \"kubernetes.io/projected/4bb3e3ca-18ce-4d77-9150-30ca3adfc11f-kube-api-access-dxgrg\") pod \"periodic-gathering-stq6m-cr8z8\" (UID: \"4bb3e3ca-18ce-4d77-9150-30ca3adfc11f\") " pod="openshift-insights/periodic-gathering-stq6m-cr8z8"
Oct 30 20:39:59.381011 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:59.380968    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9241b135-3961-4ef5-9fcd-96a078e2475f-console-config\") pod \"console-6cf8db96c7-kw294\" (UID: \"9241b135-3961-4ef5-9fcd-96a078e2475f\") " pod="openshift-console/console-6cf8db96c7-kw294"
Oct 30 20:39:59.381011 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:59.380993    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9241b135-3961-4ef5-9fcd-96a078e2475f-oauth-serving-cert\") pod \"console-6cf8db96c7-kw294\" (UID: \"9241b135-3961-4ef5-9fcd-96a078e2475f\") " pod="openshift-console/console-6cf8db96c7-kw294"
Oct 30 20:39:59.381317 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:59.381175    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9241b135-3961-4ef5-9fcd-96a078e2475f-trusted-ca-bundle\") pod \"console-6cf8db96c7-kw294\" (UID: \"9241b135-3961-4ef5-9fcd-96a078e2475f\") " pod="openshift-console/console-6cf8db96c7-kw294"
Oct 30 20:39:59.381317 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:59.381219    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9241b135-3961-4ef5-9fcd-96a078e2475f-console-serving-cert\") pod \"console-6cf8db96c7-kw294\" (UID: \"9241b135-3961-4ef5-9fcd-96a078e2475f\") " pod="openshift-console/console-6cf8db96c7-kw294"
Oct 30 20:39:59.381317 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:59.381168    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9033c4e2-4586-4ec9-996b-21c412cfb8ba-archives-path" pod "9033c4e2-4586-4ec9-996b-21c412cfb8ba" (UID: "9033c4e2-4586-4ec9-996b-21c412cfb8ba"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:39:59.381317 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:59.381217    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9033c4e2-4586-4ec9-996b-21c412cfb8ba-trusted-ca-bundle" pod "9033c4e2-4586-4ec9-996b-21c412cfb8ba" (UID: "9033c4e2-4586-4ec9-996b-21c412cfb8ba"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:39:59.381317 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:59.381237    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9033c4e2-4586-4ec9-996b-21c412cfb8ba-service-ca-bundle" pod "9033c4e2-4586-4ec9-996b-21c412cfb8ba" (UID: "9033c4e2-4586-4ec9-996b-21c412cfb8ba"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:39:59.381317 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:59.381296    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9241b135-3961-4ef5-9fcd-96a078e2475f-console-oauth-config\") pod \"console-6cf8db96c7-kw294\" (UID: \"9241b135-3961-4ef5-9fcd-96a078e2475f\") " pod="openshift-console/console-6cf8db96c7-kw294"
Oct 30 20:39:59.381621 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:59.381364    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9241b135-3961-4ef5-9fcd-96a078e2475f-tmp\") pod \"console-6cf8db96c7-kw294\" (UID: \"9241b135-3961-4ef5-9fcd-96a078e2475f\") " pod="openshift-console/console-6cf8db96c7-kw294"
Oct 30 20:39:59.381621 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:59.381416    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4bb3e3ca-18ce-4d77-9150-30ca3adfc11f-trusted-ca-bundle\") pod \"periodic-gathering-stq6m-cr8z8\" (UID: \"4bb3e3ca-18ce-4d77-9150-30ca3adfc11f\") " pod="openshift-insights/periodic-gathering-stq6m-cr8z8"
Oct 30 20:39:59.381621 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:59.381444    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9241b135-3961-4ef5-9fcd-96a078e2475f-service-ca\") pod \"console-6cf8db96c7-kw294\" (UID: \"9241b135-3961-4ef5-9fcd-96a078e2475f\") " pod="openshift-console/console-6cf8db96c7-kw294"
Oct 30 20:39:59.381621 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:59.381474    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/4bb3e3ca-18ce-4d77-9150-30ca3adfc11f-archives-path\") pod \"periodic-gathering-stq6m-cr8z8\" (UID: \"4bb3e3ca-18ce-4d77-9150-30ca3adfc11f\") " pod="openshift-insights/periodic-gathering-stq6m-cr8z8"
Oct 30 20:39:59.381621 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:59.381504    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4bb3e3ca-18ce-4d77-9150-30ca3adfc11f-service-ca-bundle\") pod \"periodic-gathering-stq6m-cr8z8\" (UID: \"4bb3e3ca-18ce-4d77-9150-30ca3adfc11f\") " pod="openshift-insights/periodic-gathering-stq6m-cr8z8"
Oct 30 20:39:59.381621 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:59.381535    4521 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9033c4e2-4586-4ec9-996b-21c412cfb8ba-trusted-ca-bundle\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:39:59.381621 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:59.381547    4521 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/9033c4e2-4586-4ec9-996b-21c412cfb8ba-archives-path\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:39:59.381621 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:59.381558    4521 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9033c4e2-4586-4ec9-996b-21c412cfb8ba-service-ca-bundle\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:39:59.382006 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:59.381828    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/4bb3e3ca-18ce-4d77-9150-30ca3adfc11f-archives-path\") pod \"periodic-gathering-stq6m-cr8z8\" (UID: \"4bb3e3ca-18ce-4d77-9150-30ca3adfc11f\") " pod="openshift-insights/periodic-gathering-stq6m-cr8z8"
Oct 30 20:39:59.382277 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:59.382251    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4bb3e3ca-18ce-4d77-9150-30ca3adfc11f-service-ca-bundle\") pod \"periodic-gathering-stq6m-cr8z8\" (UID: \"4bb3e3ca-18ce-4d77-9150-30ca3adfc11f\") " pod="openshift-insights/periodic-gathering-stq6m-cr8z8"
Oct 30 20:39:59.382472 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:59.382453    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4bb3e3ca-18ce-4d77-9150-30ca3adfc11f-trusted-ca-bundle\") pod \"periodic-gathering-stq6m-cr8z8\" (UID: \"4bb3e3ca-18ce-4d77-9150-30ca3adfc11f\") " pod="openshift-insights/periodic-gathering-stq6m-cr8z8"
Oct 30 20:39:59.383257 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:59.383238    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9033c4e2-4586-4ec9-996b-21c412cfb8ba-kube-api-access-2lvnm" pod "9033c4e2-4586-4ec9-996b-21c412cfb8ba" (UID: "9033c4e2-4586-4ec9-996b-21c412cfb8ba"). InnerVolumeSpecName "kube-api-access-2lvnm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:39:59.394350 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:59.394314    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxgrg\" (UniqueName: \"kubernetes.io/projected/4bb3e3ca-18ce-4d77-9150-30ca3adfc11f-kube-api-access-dxgrg\") pod \"periodic-gathering-stq6m-cr8z8\" (UID: \"4bb3e3ca-18ce-4d77-9150-30ca3adfc11f\") " pod="openshift-insights/periodic-gathering-stq6m-cr8z8"
Oct 30 20:39:59.482640 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:59.482595    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9241b135-3961-4ef5-9fcd-96a078e2475f-trusted-ca-bundle\") pod \"console-6cf8db96c7-kw294\" (UID: \"9241b135-3961-4ef5-9fcd-96a078e2475f\") " pod="openshift-console/console-6cf8db96c7-kw294"
Oct 30 20:39:59.482640 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:59.482640    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9241b135-3961-4ef5-9fcd-96a078e2475f-console-serving-cert\") pod \"console-6cf8db96c7-kw294\" (UID: \"9241b135-3961-4ef5-9fcd-96a078e2475f\") " pod="openshift-console/console-6cf8db96c7-kw294"
Oct 30 20:39:59.482900 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:59.482658    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9241b135-3961-4ef5-9fcd-96a078e2475f-console-oauth-config\") pod \"console-6cf8db96c7-kw294\" (UID: \"9241b135-3961-4ef5-9fcd-96a078e2475f\") " pod="openshift-console/console-6cf8db96c7-kw294"
Oct 30 20:39:59.482900 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:59.482673    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9241b135-3961-4ef5-9fcd-96a078e2475f-tmp\") pod \"console-6cf8db96c7-kw294\" (UID: \"9241b135-3961-4ef5-9fcd-96a078e2475f\") " pod="openshift-console/console-6cf8db96c7-kw294"
Oct 30 20:39:59.482900 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:59.482711    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9241b135-3961-4ef5-9fcd-96a078e2475f-service-ca\") pod \"console-6cf8db96c7-kw294\" (UID: \"9241b135-3961-4ef5-9fcd-96a078e2475f\") " pod="openshift-console/console-6cf8db96c7-kw294"
Oct 30 20:39:59.482900 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:59.482753    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gqlh\" (UniqueName: \"kubernetes.io/projected/9241b135-3961-4ef5-9fcd-96a078e2475f-kube-api-access-9gqlh\") pod \"console-6cf8db96c7-kw294\" (UID: \"9241b135-3961-4ef5-9fcd-96a078e2475f\") " pod="openshift-console/console-6cf8db96c7-kw294"
Oct 30 20:39:59.482900 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:59.482862    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9241b135-3961-4ef5-9fcd-96a078e2475f-console-config\") pod \"console-6cf8db96c7-kw294\" (UID: \"9241b135-3961-4ef5-9fcd-96a078e2475f\") " pod="openshift-console/console-6cf8db96c7-kw294"
Oct 30 20:39:59.482900 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:59.482900    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9241b135-3961-4ef5-9fcd-96a078e2475f-oauth-serving-cert\") pod \"console-6cf8db96c7-kw294\" (UID: \"9241b135-3961-4ef5-9fcd-96a078e2475f\") " pod="openshift-console/console-6cf8db96c7-kw294"
Oct 30 20:39:59.483231 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:59.482963    4521 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2lvnm\" (UniqueName: \"kubernetes.io/projected/9033c4e2-4586-4ec9-996b-21c412cfb8ba-kube-api-access-2lvnm\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:39:59.483355 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:59.483315    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9241b135-3961-4ef5-9fcd-96a078e2475f-tmp\") pod \"console-6cf8db96c7-kw294\" (UID: \"9241b135-3961-4ef5-9fcd-96a078e2475f\") " pod="openshift-console/console-6cf8db96c7-kw294"
Oct 30 20:39:59.483638 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:59.483616    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9241b135-3961-4ef5-9fcd-96a078e2475f-oauth-serving-cert\") pod \"console-6cf8db96c7-kw294\" (UID: \"9241b135-3961-4ef5-9fcd-96a078e2475f\") " pod="openshift-console/console-6cf8db96c7-kw294"
Oct 30 20:39:59.483713 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:59.483666    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9241b135-3961-4ef5-9fcd-96a078e2475f-trusted-ca-bundle\") pod \"console-6cf8db96c7-kw294\" (UID: \"9241b135-3961-4ef5-9fcd-96a078e2475f\") " pod="openshift-console/console-6cf8db96c7-kw294"
Oct 30 20:39:59.483713 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:59.483677    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9241b135-3961-4ef5-9fcd-96a078e2475f-service-ca\") pod \"console-6cf8db96c7-kw294\" (UID: \"9241b135-3961-4ef5-9fcd-96a078e2475f\") " pod="openshift-console/console-6cf8db96c7-kw294"
Oct 30 20:39:59.483835 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:59.483803    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9241b135-3961-4ef5-9fcd-96a078e2475f-console-config\") pod \"console-6cf8db96c7-kw294\" (UID: \"9241b135-3961-4ef5-9fcd-96a078e2475f\") " pod="openshift-console/console-6cf8db96c7-kw294"
Oct 30 20:39:59.485160 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:59.485138    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9241b135-3961-4ef5-9fcd-96a078e2475f-console-oauth-config\") pod \"console-6cf8db96c7-kw294\" (UID: \"9241b135-3961-4ef5-9fcd-96a078e2475f\") " pod="openshift-console/console-6cf8db96c7-kw294"
Oct 30 20:39:59.485253 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:59.485237    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9241b135-3961-4ef5-9fcd-96a078e2475f-console-serving-cert\") pod \"console-6cf8db96c7-kw294\" (UID: \"9241b135-3961-4ef5-9fcd-96a078e2475f\") " pod="openshift-console/console-6cf8db96c7-kw294"
Oct 30 20:39:59.495114 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:59.495092    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gqlh\" (UniqueName: \"kubernetes.io/projected/9241b135-3961-4ef5-9fcd-96a078e2475f-kube-api-access-9gqlh\") pod \"console-6cf8db96c7-kw294\" (UID: \"9241b135-3961-4ef5-9fcd-96a078e2475f\") " pod="openshift-console/console-6cf8db96c7-kw294"
Oct 30 20:39:59.518729 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:59.518695    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-stq6m-cr8z8"
Oct 30 20:39:59.626983 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:59.626948    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6cf8db96c7-kw294"
Oct 30 20:39:59.814444 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:59.814374    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-stq6m-cr8z8"]
Oct 30 20:39:59.817225 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:39:59.817192    4521 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bb3e3ca_18ce_4d77_9150_30ca3adfc11f.slice/crio-3dc242d1f33b30cc5d8c990d547e4c63a880d9e42d1ede8817789e821428de69 WatchSource:0}: Error finding container 3dc242d1f33b30cc5d8c990d547e4c63a880d9e42d1ede8817789e821428de69: Status 404 returned error can't find the container with id 3dc242d1f33b30cc5d8c990d547e4c63a880d9e42d1ede8817789e821428de69
Oct 30 20:39:59.925345 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:39:59.925304    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6cf8db96c7-kw294"]
Oct 30 20:39:59.928278 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:39:59.928249    4521 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9241b135_3961_4ef5_9fcd_96a078e2475f.slice/crio-c86a47a238ef72f99230a6ba2bdfa84ad8194cac2bd42cd4d226de44eb42dbf5 WatchSource:0}: Error finding container c86a47a238ef72f99230a6ba2bdfa84ad8194cac2bd42cd4d226de44eb42dbf5: Status 404 returned error can't find the container with id c86a47a238ef72f99230a6ba2bdfa84ad8194cac2bd42cd4d226de44eb42dbf5
Oct 30 20:40:00.084033 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:40:00.083400    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6cf8db96c7-kw294" event={"ID":"9241b135-3961-4ef5-9fcd-96a078e2475f","Type":"ContainerStarted","Data":"47d5eef55bcf604b2d886a0e790ec71bac0b91392921b2d6efb8fe36f85dd0e7"}
Oct 30 20:40:00.084033 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:40:00.083454    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6cf8db96c7-kw294" event={"ID":"9241b135-3961-4ef5-9fcd-96a078e2475f","Type":"ContainerStarted","Data":"c86a47a238ef72f99230a6ba2bdfa84ad8194cac2bd42cd4d226de44eb42dbf5"}
Oct 30 20:40:00.086604 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:40:00.086562    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-stq6m-cr8z8" event={"ID":"4bb3e3ca-18ce-4d77-9150-30ca3adfc11f","Type":"ContainerStarted","Data":"ad8e04bdf8e02e6da546e83833fff566a9912d041f5b97aa748ed32c0eeb8fcc"}
Oct 30 20:40:00.086604 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:40:00.086609    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-stq6m-cr8z8" event={"ID":"4bb3e3ca-18ce-4d77-9150-30ca3adfc11f","Type":"ContainerStarted","Data":"3dc242d1f33b30cc5d8c990d547e4c63a880d9e42d1ede8817789e821428de69"}
Oct 30 20:40:00.088013 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:40:00.087979    4521 generic.go:356] "Generic (PLEG): container finished" podID="9033c4e2-4586-4ec9-996b-21c412cfb8ba" containerID="84f75d512bda86246800d98992feae6f917091e0b59369751f878ee85d0ae742" exitCode=2
Oct 30 20:40:00.088283 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:40:00.088262    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-4f444-dttkw"
Oct 30 20:40:00.088407 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:40:00.088286    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-4f444-dttkw" event={"ID":"9033c4e2-4586-4ec9-996b-21c412cfb8ba","Type":"ContainerDied","Data":"84f75d512bda86246800d98992feae6f917091e0b59369751f878ee85d0ae742"}
Oct 30 20:40:00.088407 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:40:00.088346    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-4f444-dttkw" event={"ID":"9033c4e2-4586-4ec9-996b-21c412cfb8ba","Type":"ContainerDied","Data":"958c9bd7c6d62bcfd6effc523bd6f0835c3ba9f68387a4e1770cf0efdf7ae66b"}
Oct 30 20:40:00.088407 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:40:00.088388    4521 scope.go:122] "RemoveContainer" containerID="84f75d512bda86246800d98992feae6f917091e0b59369751f878ee85d0ae742"
Oct 30 20:40:00.097773 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:40:00.097698    4521 scope.go:122] "RemoveContainer" containerID="84f75d512bda86246800d98992feae6f917091e0b59369751f878ee85d0ae742"
Oct 30 20:40:00.098068 ip-10-0-131-125 kubenswrapper[4521]: E1030 20:40:00.098036    4521 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84f75d512bda86246800d98992feae6f917091e0b59369751f878ee85d0ae742\": container with ID starting with 84f75d512bda86246800d98992feae6f917091e0b59369751f878ee85d0ae742 not found: ID does not exist" containerID="84f75d512bda86246800d98992feae6f917091e0b59369751f878ee85d0ae742"
Oct 30 20:40:00.098131 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:40:00.098065    4521 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84f75d512bda86246800d98992feae6f917091e0b59369751f878ee85d0ae742"} err="failed to get container status \"84f75d512bda86246800d98992feae6f917091e0b59369751f878ee85d0ae742\": rpc error: code = NotFound desc = could not find container \"84f75d512bda86246800d98992feae6f917091e0b59369751f878ee85d0ae742\": container with ID starting with 84f75d512bda86246800d98992feae6f917091e0b59369751f878ee85d0ae742 not found: ID does not exist"
Oct 30 20:40:00.105928 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:40:00.105878    4521 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-6cf8db96c7-kw294" podStartSLOduration=1.1058628 podStartE2EDuration="1.1058628s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:39:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:40:00.105184268 +0000 UTC m=+67.450805506" watchObservedRunningTime="2026-10-30 20:40:00.1058628 +0000 UTC m=+67.451484037"
Oct 30 20:40:00.118136 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:40:00.118103    4521 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-4f444-dttkw"]
Oct 30 20:40:00.122417 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:40:00.122391    4521 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-4f444-dttkw"]
Oct 30 20:40:00.139810 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:40:00.139759    4521 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-stq6m-cr8z8" podStartSLOduration=1.139745864 podStartE2EDuration="1.139745864s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:39:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:40:00.138652912 +0000 UTC m=+67.484274151" watchObservedRunningTime="2026-10-30 20:40:00.139745864 +0000 UTC m=+67.485367102"
Oct 30 20:40:00.789768 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:40:00.789736    4521 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9033c4e2-4586-4ec9-996b-21c412cfb8ba" path="/var/lib/kubelet/pods/9033c4e2-4586-4ec9-996b-21c412cfb8ba/volumes"
Oct 30 20:40:09.627248 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:40:09.627205    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6cf8db96c7-kw294"
Oct 30 20:40:09.627777 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:40:09.627271    4521 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6cf8db96c7-kw294"
Oct 30 20:40:09.632008 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:40:09.631988    4521 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6cf8db96c7-kw294"
Oct 30 20:40:10.125367 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:40:10.125336    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6cf8db96c7-kw294"
Oct 30 20:40:10.177537 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:40:10.177508    4521 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8d77f6bfd-hr84v"]
Oct 30 20:40:16.428291 ip-10-0-131-125 kubenswrapper[4521]: E1030 20:40:16.428253    4521 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lkn2k" containerName="extractor"
Oct 30 20:40:24.187112 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:40:24.187035    4521 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-74cc97464-lhsft" podUID="e3915ee4-ed6d-43e1-816c-b33e05bf78e0" containerName="console" containerID="cri-o://9da7676dc8fd57ccd1c45183a3874f8b9c5ab699aa86c0ddd05fdc581287e1a4" gracePeriod=15
Oct 30 20:40:24.448634 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:40:24.448563    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74cc97464-lhsft"
Oct 30 20:40:24.585866 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:40:24.585827    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3915ee4-ed6d-43e1-816c-b33e05bf78e0-tmp\" (UniqueName: \"kubernetes.io/empty-dir/e3915ee4-ed6d-43e1-816c-b33e05bf78e0-tmp\") pod \"e3915ee4-ed6d-43e1-816c-b33e05bf78e0\" (UID: \"e3915ee4-ed6d-43e1-816c-b33e05bf78e0\") "
Oct 30 20:40:24.585866 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:40:24.585869    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3915ee4-ed6d-43e1-816c-b33e05bf78e0-console-config\" (UniqueName: \"kubernetes.io/configmap/e3915ee4-ed6d-43e1-816c-b33e05bf78e0-console-config\") pod \"e3915ee4-ed6d-43e1-816c-b33e05bf78e0\" (UID: \"e3915ee4-ed6d-43e1-816c-b33e05bf78e0\") "
Oct 30 20:40:24.586118 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:40:24.585888    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3915ee4-ed6d-43e1-816c-b33e05bf78e0-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e3915ee4-ed6d-43e1-816c-b33e05bf78e0-oauth-serving-cert\") pod \"e3915ee4-ed6d-43e1-816c-b33e05bf78e0\" (UID: \"e3915ee4-ed6d-43e1-816c-b33e05bf78e0\") "
Oct 30 20:40:24.586118 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:40:24.585920    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e3915ee4-ed6d-43e1-816c-b33e05bf78e0-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e3915ee4-ed6d-43e1-816c-b33e05bf78e0-console-serving-cert\") pod \"e3915ee4-ed6d-43e1-816c-b33e05bf78e0\" (UID: \"e3915ee4-ed6d-43e1-816c-b33e05bf78e0\") "
Oct 30 20:40:24.586118 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:40:24.585946    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3915ee4-ed6d-43e1-816c-b33e05bf78e0-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3915ee4-ed6d-43e1-816c-b33e05bf78e0-service-ca\") pod \"e3915ee4-ed6d-43e1-816c-b33e05bf78e0\" (UID: \"e3915ee4-ed6d-43e1-816c-b33e05bf78e0\") "
Oct 30 20:40:24.586118 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:40:24.585995    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e3915ee4-ed6d-43e1-816c-b33e05bf78e0-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e3915ee4-ed6d-43e1-816c-b33e05bf78e0-console-oauth-config\") pod \"e3915ee4-ed6d-43e1-816c-b33e05bf78e0\" (UID: \"e3915ee4-ed6d-43e1-816c-b33e05bf78e0\") "
Oct 30 20:40:24.586118 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:40:24.586041    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3915ee4-ed6d-43e1-816c-b33e05bf78e0-kube-api-access-9nmv2\" (UniqueName: \"kubernetes.io/projected/e3915ee4-ed6d-43e1-816c-b33e05bf78e0-kube-api-access-9nmv2\") pod \"e3915ee4-ed6d-43e1-816c-b33e05bf78e0\" (UID: \"e3915ee4-ed6d-43e1-816c-b33e05bf78e0\") "
Oct 30 20:40:24.586385 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:40:24.586220    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3915ee4-ed6d-43e1-816c-b33e05bf78e0-tmp" pod "e3915ee4-ed6d-43e1-816c-b33e05bf78e0" (UID: "e3915ee4-ed6d-43e1-816c-b33e05bf78e0"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:40:24.586385 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:40:24.586304    4521 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e3915ee4-ed6d-43e1-816c-b33e05bf78e0-tmp\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:40:24.586490 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:40:24.586427    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3915ee4-ed6d-43e1-816c-b33e05bf78e0-oauth-serving-cert" pod "e3915ee4-ed6d-43e1-816c-b33e05bf78e0" (UID: "e3915ee4-ed6d-43e1-816c-b33e05bf78e0"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:40:24.586490 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:40:24.586441    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3915ee4-ed6d-43e1-816c-b33e05bf78e0-service-ca" pod "e3915ee4-ed6d-43e1-816c-b33e05bf78e0" (UID: "e3915ee4-ed6d-43e1-816c-b33e05bf78e0"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:40:24.586585 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:40:24.586534    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3915ee4-ed6d-43e1-816c-b33e05bf78e0-console-config" pod "e3915ee4-ed6d-43e1-816c-b33e05bf78e0" (UID: "e3915ee4-ed6d-43e1-816c-b33e05bf78e0"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:40:24.588365 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:40:24.588335    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3915ee4-ed6d-43e1-816c-b33e05bf78e0-console-oauth-config" pod "e3915ee4-ed6d-43e1-816c-b33e05bf78e0" (UID: "e3915ee4-ed6d-43e1-816c-b33e05bf78e0"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:40:24.588365 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:40:24.588355    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3915ee4-ed6d-43e1-816c-b33e05bf78e0-kube-api-access-9nmv2" pod "e3915ee4-ed6d-43e1-816c-b33e05bf78e0" (UID: "e3915ee4-ed6d-43e1-816c-b33e05bf78e0"). InnerVolumeSpecName "kube-api-access-9nmv2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:40:24.588515 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:40:24.588351    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3915ee4-ed6d-43e1-816c-b33e05bf78e0-console-serving-cert" pod "e3915ee4-ed6d-43e1-816c-b33e05bf78e0" (UID: "e3915ee4-ed6d-43e1-816c-b33e05bf78e0"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:40:24.686905 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:40:24.686865    4521 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e3915ee4-ed6d-43e1-816c-b33e05bf78e0-console-config\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:40:24.686905 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:40:24.686898    4521 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e3915ee4-ed6d-43e1-816c-b33e05bf78e0-oauth-serving-cert\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:40:24.686905 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:40:24.686909    4521 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e3915ee4-ed6d-43e1-816c-b33e05bf78e0-console-serving-cert\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:40:24.687135 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:40:24.686918    4521 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e3915ee4-ed6d-43e1-816c-b33e05bf78e0-service-ca\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:40:24.687135 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:40:24.686926    4521 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e3915ee4-ed6d-43e1-816c-b33e05bf78e0-console-oauth-config\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:40:24.687135 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:40:24.686935    4521 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9nmv2\" (UniqueName: \"kubernetes.io/projected/e3915ee4-ed6d-43e1-816c-b33e05bf78e0-kube-api-access-9nmv2\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:40:25.174558 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:40:25.174465    4521 generic.go:356] "Generic (PLEG): container finished" podID="e3915ee4-ed6d-43e1-816c-b33e05bf78e0" containerID="9da7676dc8fd57ccd1c45183a3874f8b9c5ab699aa86c0ddd05fdc581287e1a4" exitCode=0
Oct 30 20:40:25.174558 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:40:25.174508    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74cc97464-lhsft" event={"ID":"e3915ee4-ed6d-43e1-816c-b33e05bf78e0","Type":"ContainerDied","Data":"9da7676dc8fd57ccd1c45183a3874f8b9c5ab699aa86c0ddd05fdc581287e1a4"}
Oct 30 20:40:25.174558 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:40:25.174515    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74cc97464-lhsft"
Oct 30 20:40:25.174558 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:40:25.174555    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74cc97464-lhsft" event={"ID":"e3915ee4-ed6d-43e1-816c-b33e05bf78e0","Type":"ContainerDied","Data":"b853daa581d07a7cb29356e1c07e57dbdd698575b23d627be749b5fbb78edfa6"}
Oct 30 20:40:25.174935 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:40:25.174578    4521 scope.go:122] "RemoveContainer" containerID="9da7676dc8fd57ccd1c45183a3874f8b9c5ab699aa86c0ddd05fdc581287e1a4"
Oct 30 20:40:25.182332 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:40:25.182302    4521 scope.go:122] "RemoveContainer" containerID="9da7676dc8fd57ccd1c45183a3874f8b9c5ab699aa86c0ddd05fdc581287e1a4"
Oct 30 20:40:25.182598 ip-10-0-131-125 kubenswrapper[4521]: E1030 20:40:25.182576    4521 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9da7676dc8fd57ccd1c45183a3874f8b9c5ab699aa86c0ddd05fdc581287e1a4\": container with ID starting with 9da7676dc8fd57ccd1c45183a3874f8b9c5ab699aa86c0ddd05fdc581287e1a4 not found: ID does not exist" containerID="9da7676dc8fd57ccd1c45183a3874f8b9c5ab699aa86c0ddd05fdc581287e1a4"
Oct 30 20:40:25.182690 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:40:25.182604    4521 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9da7676dc8fd57ccd1c45183a3874f8b9c5ab699aa86c0ddd05fdc581287e1a4"} err="failed to get container status \"9da7676dc8fd57ccd1c45183a3874f8b9c5ab699aa86c0ddd05fdc581287e1a4\": rpc error: code = NotFound desc = could not find container \"9da7676dc8fd57ccd1c45183a3874f8b9c5ab699aa86c0ddd05fdc581287e1a4\": container with ID starting with 9da7676dc8fd57ccd1c45183a3874f8b9c5ab699aa86c0ddd05fdc581287e1a4 not found: ID does not exist"
Oct 30 20:40:25.192178 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:40:25.192155    4521 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-74cc97464-lhsft"]
Oct 30 20:40:25.196553 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:40:25.196530    4521 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-74cc97464-lhsft"]
Oct 30 20:40:26.789637 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:40:26.789605    4521 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3915ee4-ed6d-43e1-816c-b33e05bf78e0" path="/var/lib/kubelet/pods/e3915ee4-ed6d-43e1-816c-b33e05bf78e0/volumes"
Oct 30 20:40:35.197594 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:40:35.197552    4521 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-8d77f6bfd-hr84v" podUID="bf502e09-d489-4574-83e9-98232c580449" containerName="console" containerID="cri-o://abf99511ab7e13234638e624526730678b8c3668b176ca51f38ebe144a87c989" gracePeriod=15
Oct 30 20:40:35.432083 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:40:35.432057    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8d77f6bfd-hr84v"
Oct 30 20:40:35.571146 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:40:35.571049    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf502e09-d489-4574-83e9-98232c580449-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bf502e09-d489-4574-83e9-98232c580449-trusted-ca-bundle\") pod \"bf502e09-d489-4574-83e9-98232c580449\" (UID: \"bf502e09-d489-4574-83e9-98232c580449\") "
Oct 30 20:40:35.571146 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:40:35.571098    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf502e09-d489-4574-83e9-98232c580449-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bf502e09-d489-4574-83e9-98232c580449-console-oauth-config\") pod \"bf502e09-d489-4574-83e9-98232c580449\" (UID: \"bf502e09-d489-4574-83e9-98232c580449\") "
Oct 30 20:40:35.571146 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:40:35.571122    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf502e09-d489-4574-83e9-98232c580449-kube-api-access-nkr29\" (UniqueName: \"kubernetes.io/projected/bf502e09-d489-4574-83e9-98232c580449-kube-api-access-nkr29\") pod \"bf502e09-d489-4574-83e9-98232c580449\" (UID: \"bf502e09-d489-4574-83e9-98232c580449\") "
Oct 30 20:40:35.571451 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:40:35.571155    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf502e09-d489-4574-83e9-98232c580449-console-config\" (UniqueName: \"kubernetes.io/configmap/bf502e09-d489-4574-83e9-98232c580449-console-config\") pod \"bf502e09-d489-4574-83e9-98232c580449\" (UID: \"bf502e09-d489-4574-83e9-98232c580449\") "
Oct 30 20:40:35.571451 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:40:35.571273    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf502e09-d489-4574-83e9-98232c580449-tmp\" (UniqueName: \"kubernetes.io/empty-dir/bf502e09-d489-4574-83e9-98232c580449-tmp\") pod \"bf502e09-d489-4574-83e9-98232c580449\" (UID: \"bf502e09-d489-4574-83e9-98232c580449\") "
Oct 30 20:40:35.571451 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:40:35.571349    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf502e09-d489-4574-83e9-98232c580449-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bf502e09-d489-4574-83e9-98232c580449-console-serving-cert\") pod \"bf502e09-d489-4574-83e9-98232c580449\" (UID: \"bf502e09-d489-4574-83e9-98232c580449\") "
Oct 30 20:40:35.571451 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:40:35.571435    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf502e09-d489-4574-83e9-98232c580449-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bf502e09-d489-4574-83e9-98232c580449-oauth-serving-cert\") pod \"bf502e09-d489-4574-83e9-98232c580449\" (UID: \"bf502e09-d489-4574-83e9-98232c580449\") "
Oct 30 20:40:35.571641 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:40:35.571478    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf502e09-d489-4574-83e9-98232c580449-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf502e09-d489-4574-83e9-98232c580449-service-ca\") pod \"bf502e09-d489-4574-83e9-98232c580449\" (UID: \"bf502e09-d489-4574-83e9-98232c580449\") "
Oct 30 20:40:35.571641 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:40:35.571616    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf502e09-d489-4574-83e9-98232c580449-trusted-ca-bundle" pod "bf502e09-d489-4574-83e9-98232c580449" (UID: "bf502e09-d489-4574-83e9-98232c580449"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:40:35.571749 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:40:35.571641    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf502e09-d489-4574-83e9-98232c580449-tmp" pod "bf502e09-d489-4574-83e9-98232c580449" (UID: "bf502e09-d489-4574-83e9-98232c580449"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:40:35.571749 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:40:35.571681    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf502e09-d489-4574-83e9-98232c580449-console-config" pod "bf502e09-d489-4574-83e9-98232c580449" (UID: "bf502e09-d489-4574-83e9-98232c580449"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:40:35.571749 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:40:35.571712    4521 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bf502e09-d489-4574-83e9-98232c580449-trusted-ca-bundle\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:40:35.571955 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:40:35.571935    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf502e09-d489-4574-83e9-98232c580449-oauth-serving-cert" pod "bf502e09-d489-4574-83e9-98232c580449" (UID: "bf502e09-d489-4574-83e9-98232c580449"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:40:35.572010 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:40:35.571958    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf502e09-d489-4574-83e9-98232c580449-service-ca" pod "bf502e09-d489-4574-83e9-98232c580449" (UID: "bf502e09-d489-4574-83e9-98232c580449"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:40:35.573281 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:40:35.573260    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf502e09-d489-4574-83e9-98232c580449-console-oauth-config" pod "bf502e09-d489-4574-83e9-98232c580449" (UID: "bf502e09-d489-4574-83e9-98232c580449"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:40:35.573766 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:40:35.573739    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf502e09-d489-4574-83e9-98232c580449-console-serving-cert" pod "bf502e09-d489-4574-83e9-98232c580449" (UID: "bf502e09-d489-4574-83e9-98232c580449"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:40:35.573859 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:40:35.573763    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf502e09-d489-4574-83e9-98232c580449-kube-api-access-nkr29" pod "bf502e09-d489-4574-83e9-98232c580449" (UID: "bf502e09-d489-4574-83e9-98232c580449"). InnerVolumeSpecName "kube-api-access-nkr29". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:40:35.672928 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:40:35.672887    4521 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bf502e09-d489-4574-83e9-98232c580449-oauth-serving-cert\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:40:35.672928 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:40:35.672918    4521 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bf502e09-d489-4574-83e9-98232c580449-service-ca\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:40:35.672928 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:40:35.672928    4521 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bf502e09-d489-4574-83e9-98232c580449-console-oauth-config\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:40:35.672928 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:40:35.672936    4521 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nkr29\" (UniqueName: \"kubernetes.io/projected/bf502e09-d489-4574-83e9-98232c580449-kube-api-access-nkr29\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:40:35.673196 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:40:35.672947    4521 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bf502e09-d489-4574-83e9-98232c580449-console-config\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:40:35.673196 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:40:35.672957    4521 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bf502e09-d489-4574-83e9-98232c580449-tmp\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:40:35.673196 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:40:35.672967    4521 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bf502e09-d489-4574-83e9-98232c580449-console-serving-cert\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:40:36.212198 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:40:36.212159    4521 generic.go:356] "Generic (PLEG): container finished" podID="bf502e09-d489-4574-83e9-98232c580449" containerID="abf99511ab7e13234638e624526730678b8c3668b176ca51f38ebe144a87c989" exitCode=0
Oct 30 20:40:36.212627 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:40:36.212219    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8d77f6bfd-hr84v"
Oct 30 20:40:36.212627 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:40:36.212222    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8d77f6bfd-hr84v" event={"ID":"bf502e09-d489-4574-83e9-98232c580449","Type":"ContainerDied","Data":"abf99511ab7e13234638e624526730678b8c3668b176ca51f38ebe144a87c989"}
Oct 30 20:40:36.212627 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:40:36.212294    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8d77f6bfd-hr84v" event={"ID":"bf502e09-d489-4574-83e9-98232c580449","Type":"ContainerDied","Data":"0ec23e92252dbb24c8500aa4109a84c610c30667421574e6e7ebacabfbf13028"}
Oct 30 20:40:36.212627 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:40:36.212316    4521 scope.go:122] "RemoveContainer" containerID="abf99511ab7e13234638e624526730678b8c3668b176ca51f38ebe144a87c989"
Oct 30 20:40:36.220597 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:40:36.220577    4521 scope.go:122] "RemoveContainer" containerID="abf99511ab7e13234638e624526730678b8c3668b176ca51f38ebe144a87c989"
Oct 30 20:40:36.220869 ip-10-0-131-125 kubenswrapper[4521]: E1030 20:40:36.220848    4521 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"abf99511ab7e13234638e624526730678b8c3668b176ca51f38ebe144a87c989\": container with ID starting with abf99511ab7e13234638e624526730678b8c3668b176ca51f38ebe144a87c989 not found: ID does not exist" containerID="abf99511ab7e13234638e624526730678b8c3668b176ca51f38ebe144a87c989"
Oct 30 20:40:36.220930 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:40:36.220880    4521 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abf99511ab7e13234638e624526730678b8c3668b176ca51f38ebe144a87c989"} err="failed to get container status \"abf99511ab7e13234638e624526730678b8c3668b176ca51f38ebe144a87c989\": rpc error: code = NotFound desc = could not find container \"abf99511ab7e13234638e624526730678b8c3668b176ca51f38ebe144a87c989\": container with ID starting with abf99511ab7e13234638e624526730678b8c3668b176ca51f38ebe144a87c989 not found: ID does not exist"
Oct 30 20:40:36.233751 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:40:36.233728    4521 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8d77f6bfd-hr84v"]
Oct 30 20:40:36.237423 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:40:36.237401    4521 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-8d77f6bfd-hr84v"]
Oct 30 20:40:36.789743 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:40:36.789712    4521 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf502e09-d489-4574-83e9-98232c580449" path="/var/lib/kubelet/pods/bf502e09-d489-4574-83e9-98232c580449/volumes"
Oct 30 20:40:46.428199 ip-10-0-131-125 kubenswrapper[4521]: E1030 20:40:46.428159    4521 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lkn2k" containerName="extractor"
Oct 30 20:41:16.428050 ip-10-0-131-125 kubenswrapper[4521]: E1030 20:41:16.427970    4521 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lkn2k" containerName="extractor"
Oct 30 20:41:23.364171 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:23.364083    4521 generic.go:356] "Generic (PLEG): container finished" podID="4bb3e3ca-18ce-4d77-9150-30ca3adfc11f" containerID="ad8e04bdf8e02e6da546e83833fff566a9912d041f5b97aa748ed32c0eeb8fcc" exitCode=0
Oct 30 20:41:23.364612 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:23.364160    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-stq6m-cr8z8" event={"ID":"4bb3e3ca-18ce-4d77-9150-30ca3adfc11f","Type":"ContainerDied","Data":"ad8e04bdf8e02e6da546e83833fff566a9912d041f5b97aa748ed32c0eeb8fcc"}
Oct 30 20:41:24.489864 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:24.489835    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-stq6m-cr8z8"
Oct 30 20:41:24.513047 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:24.513017    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4bb3e3ca-18ce-4d77-9150-30ca3adfc11f-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4bb3e3ca-18ce-4d77-9150-30ca3adfc11f-trusted-ca-bundle\") pod \"4bb3e3ca-18ce-4d77-9150-30ca3adfc11f\" (UID: \"4bb3e3ca-18ce-4d77-9150-30ca3adfc11f\") "
Oct 30 20:41:24.513207 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:24.513078    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4bb3e3ca-18ce-4d77-9150-30ca3adfc11f-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4bb3e3ca-18ce-4d77-9150-30ca3adfc11f-service-ca-bundle\") pod \"4bb3e3ca-18ce-4d77-9150-30ca3adfc11f\" (UID: \"4bb3e3ca-18ce-4d77-9150-30ca3adfc11f\") "
Oct 30 20:41:24.513207 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:24.513115    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bb3e3ca-18ce-4d77-9150-30ca3adfc11f-kube-api-access-dxgrg\" (UniqueName: \"kubernetes.io/projected/4bb3e3ca-18ce-4d77-9150-30ca3adfc11f-kube-api-access-dxgrg\") pod \"4bb3e3ca-18ce-4d77-9150-30ca3adfc11f\" (UID: \"4bb3e3ca-18ce-4d77-9150-30ca3adfc11f\") "
Oct 30 20:41:24.513207 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:24.513143    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bb3e3ca-18ce-4d77-9150-30ca3adfc11f-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/4bb3e3ca-18ce-4d77-9150-30ca3adfc11f-archives-path\") pod \"4bb3e3ca-18ce-4d77-9150-30ca3adfc11f\" (UID: \"4bb3e3ca-18ce-4d77-9150-30ca3adfc11f\") "
Oct 30 20:41:24.513513 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:24.513478    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bb3e3ca-18ce-4d77-9150-30ca3adfc11f-trusted-ca-bundle" pod "4bb3e3ca-18ce-4d77-9150-30ca3adfc11f" (UID: "4bb3e3ca-18ce-4d77-9150-30ca3adfc11f"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:41:24.513592 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:24.513537    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bb3e3ca-18ce-4d77-9150-30ca3adfc11f-archives-path" pod "4bb3e3ca-18ce-4d77-9150-30ca3adfc11f" (UID: "4bb3e3ca-18ce-4d77-9150-30ca3adfc11f"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:41:24.513592 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:24.513542    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bb3e3ca-18ce-4d77-9150-30ca3adfc11f-service-ca-bundle" pod "4bb3e3ca-18ce-4d77-9150-30ca3adfc11f" (UID: "4bb3e3ca-18ce-4d77-9150-30ca3adfc11f"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:41:24.515558 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:24.515537    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bb3e3ca-18ce-4d77-9150-30ca3adfc11f-kube-api-access-dxgrg" pod "4bb3e3ca-18ce-4d77-9150-30ca3adfc11f" (UID: "4bb3e3ca-18ce-4d77-9150-30ca3adfc11f"). InnerVolumeSpecName "kube-api-access-dxgrg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:41:24.614222 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:24.614167    4521 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4bb3e3ca-18ce-4d77-9150-30ca3adfc11f-service-ca-bundle\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:41:24.614222 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:24.614216    4521 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxgrg\" (UniqueName: \"kubernetes.io/projected/4bb3e3ca-18ce-4d77-9150-30ca3adfc11f-kube-api-access-dxgrg\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:41:24.614222 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:24.614231    4521 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/4bb3e3ca-18ce-4d77-9150-30ca3adfc11f-archives-path\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:41:24.614494 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:24.614246    4521 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4bb3e3ca-18ce-4d77-9150-30ca3adfc11f-trusted-ca-bundle\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:41:25.372219 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:25.372190    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-stq6m-cr8z8"
Oct 30 20:41:25.372409 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:25.372191    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-stq6m-cr8z8" event={"ID":"4bb3e3ca-18ce-4d77-9150-30ca3adfc11f","Type":"ContainerDied","Data":"3dc242d1f33b30cc5d8c990d547e4c63a880d9e42d1ede8817789e821428de69"}
Oct 30 20:41:25.372409 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:25.372305    4521 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3dc242d1f33b30cc5d8c990d547e4c63a880d9e42d1ede8817789e821428de69"
Oct 30 20:41:46.428079 ip-10-0-131-125 kubenswrapper[4521]: E1030 20:41:46.428039    4521 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lkn2k" containerName="extractor"
Oct 30 20:41:47.714296 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:47.714261    4521 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644txvp8"]
Oct 30 20:41:47.714690 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:47.714633    4521 cpu_manager.go:414] "containerMap: removing container" podUID="4bb3e3ca-18ce-4d77-9150-30ca3adfc11f" containerName="insights-gathering"
Oct 30 20:41:47.714690 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:47.714650    4521 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bb3e3ca-18ce-4d77-9150-30ca3adfc11f" containerName="insights-gathering"
Oct 30 20:41:47.714690 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:47.714669    4521 cpu_manager.go:414] "containerMap: removing container" podUID="e3915ee4-ed6d-43e1-816c-b33e05bf78e0" containerName="console"
Oct 30 20:41:47.714690 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:47.714677    4521 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3915ee4-ed6d-43e1-816c-b33e05bf78e0" containerName="console"
Oct 30 20:41:47.714690 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:47.714689    4521 cpu_manager.go:414] "containerMap: removing container" podUID="bf502e09-d489-4574-83e9-98232c580449" containerName="console"
Oct 30 20:41:47.714841 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:47.714696    4521 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf502e09-d489-4574-83e9-98232c580449" containerName="console"
Oct 30 20:41:47.714841 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:47.714716    4521 cpu_manager.go:414] "containerMap: removing container" podUID="9033c4e2-4586-4ec9-996b-21c412cfb8ba" containerName="insights-gathering"
Oct 30 20:41:47.714841 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:47.714724    4521 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9033c4e2-4586-4ec9-996b-21c412cfb8ba" containerName="insights-gathering"
Oct 30 20:41:47.714841 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:47.714778    4521 memory_manager.go:365] "RemoveStaleState removing state" podUID="9033c4e2-4586-4ec9-996b-21c412cfb8ba" containerName="insights-gathering"
Oct 30 20:41:47.714841 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:47.714790    4521 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bb3e3ca-18ce-4d77-9150-30ca3adfc11f" containerName="insights-gathering"
Oct 30 20:41:47.714841 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:47.714799    4521 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3915ee4-ed6d-43e1-816c-b33e05bf78e0" containerName="console"
Oct 30 20:41:47.714841 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:47.714808    4521 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf502e09-d489-4574-83e9-98232c580449" containerName="console"
Oct 30 20:41:47.716691 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:47.716675    4521 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c984t8"]
Oct 30 20:41:47.716830 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:47.716812    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644txvp8"
Oct 30 20:41:47.718669 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:47.718647    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c984t8"
Oct 30 20:41:47.730673 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:47.730652    4521 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Oct 30 20:41:47.730822 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:47.730805    4521 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-zkfff\""
Oct 30 20:41:47.730889 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:47.730870    4521 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-bzq2j\""
Oct 30 20:41:47.737750 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:47.737721    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644txvp8"]
Oct 30 20:41:47.738632 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:47.738611    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c984t8"]
Oct 30 20:41:47.778410 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:47.778380    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1e8b2d49-82b3-47fa-8317-a5df38e00cb4-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c984t8\" (UID: \"1e8b2d49-82b3-47fa-8317-a5df38e00cb4\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c984t8"
Oct 30 20:41:47.778410 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:47.778413    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zngzx\" (UniqueName: \"kubernetes.io/projected/1e8b2d49-82b3-47fa-8317-a5df38e00cb4-kube-api-access-zngzx\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c984t8\" (UID: \"1e8b2d49-82b3-47fa-8317-a5df38e00cb4\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c984t8"
Oct 30 20:41:47.778616 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:47.778436    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/be269a25-d606-4ec1-acb4-38eedf4f201c-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644txvp8\" (UID: \"be269a25-d606-4ec1-acb4-38eedf4f201c\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644txvp8"
Oct 30 20:41:47.778616 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:47.778563    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nph6\" (UniqueName: \"kubernetes.io/projected/be269a25-d606-4ec1-acb4-38eedf4f201c-kube-api-access-6nph6\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644txvp8\" (UID: \"be269a25-d606-4ec1-acb4-38eedf4f201c\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644txvp8"
Oct 30 20:41:47.778616 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:47.778592    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1e8b2d49-82b3-47fa-8317-a5df38e00cb4-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c984t8\" (UID: \"1e8b2d49-82b3-47fa-8317-a5df38e00cb4\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c984t8"
Oct 30 20:41:47.778714 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:47.778676    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/be269a25-d606-4ec1-acb4-38eedf4f201c-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644txvp8\" (UID: \"be269a25-d606-4ec1-acb4-38eedf4f201c\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644txvp8"
Oct 30 20:41:47.823474 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:47.823438    4521 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Oct 30 20:41:47.879549 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:47.879508    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1e8b2d49-82b3-47fa-8317-a5df38e00cb4-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c984t8\" (UID: \"1e8b2d49-82b3-47fa-8317-a5df38e00cb4\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c984t8"
Oct 30 20:41:47.879549 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:47.879554    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/be269a25-d606-4ec1-acb4-38eedf4f201c-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644txvp8\" (UID: \"be269a25-d606-4ec1-acb4-38eedf4f201c\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644txvp8"
Oct 30 20:41:47.879798 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:47.879594    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1e8b2d49-82b3-47fa-8317-a5df38e00cb4-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c984t8\" (UID: \"1e8b2d49-82b3-47fa-8317-a5df38e00cb4\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c984t8"
Oct 30 20:41:47.879798 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:47.879616    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zngzx\" (UniqueName: \"kubernetes.io/projected/1e8b2d49-82b3-47fa-8317-a5df38e00cb4-kube-api-access-zngzx\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c984t8\" (UID: \"1e8b2d49-82b3-47fa-8317-a5df38e00cb4\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c984t8"
Oct 30 20:41:47.879798 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:47.879639    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/be269a25-d606-4ec1-acb4-38eedf4f201c-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644txvp8\" (UID: \"be269a25-d606-4ec1-acb4-38eedf4f201c\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644txvp8"
Oct 30 20:41:47.879798 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:47.879674    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nph6\" (UniqueName: \"kubernetes.io/projected/be269a25-d606-4ec1-acb4-38eedf4f201c-kube-api-access-6nph6\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644txvp8\" (UID: \"be269a25-d606-4ec1-acb4-38eedf4f201c\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644txvp8"
Oct 30 20:41:47.880088 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:47.879992    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1e8b2d49-82b3-47fa-8317-a5df38e00cb4-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c984t8\" (UID: \"1e8b2d49-82b3-47fa-8317-a5df38e00cb4\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c984t8"
Oct 30 20:41:47.880088 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:47.880006    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/be269a25-d606-4ec1-acb4-38eedf4f201c-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644txvp8\" (UID: \"be269a25-d606-4ec1-acb4-38eedf4f201c\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644txvp8"
Oct 30 20:41:47.880088 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:47.880047    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/be269a25-d606-4ec1-acb4-38eedf4f201c-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644txvp8\" (UID: \"be269a25-d606-4ec1-acb4-38eedf4f201c\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644txvp8"
Oct 30 20:41:47.880088 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:47.880047    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1e8b2d49-82b3-47fa-8317-a5df38e00cb4-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c984t8\" (UID: \"1e8b2d49-82b3-47fa-8317-a5df38e00cb4\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c984t8"
Oct 30 20:41:47.892130 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:47.892103    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nph6\" (UniqueName: \"kubernetes.io/projected/be269a25-d606-4ec1-acb4-38eedf4f201c-kube-api-access-6nph6\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644txvp8\" (UID: \"be269a25-d606-4ec1-acb4-38eedf4f201c\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644txvp8"
Oct 30 20:41:47.892244 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:47.892132    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zngzx\" (UniqueName: \"kubernetes.io/projected/1e8b2d49-82b3-47fa-8317-a5df38e00cb4-kube-api-access-zngzx\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c984t8\" (UID: \"1e8b2d49-82b3-47fa-8317-a5df38e00cb4\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c984t8"
Oct 30 20:41:48.029590 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:48.029492    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644txvp8"
Oct 30 20:41:48.035485 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:48.035464    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c984t8"
Oct 30 20:41:48.307309 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:48.307189    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644txvp8"]
Oct 30 20:41:48.309770 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:48.309747    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c984t8"]
Oct 30 20:41:48.310392 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:41:48.310361    4521 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe269a25_d606_4ec1_acb4_38eedf4f201c.slice/crio-aa31a4830233f3567bd5e73bc6d245157b06f95e2a3ce30a3e6ed99cdd11088d WatchSource:0}: Error finding container aa31a4830233f3567bd5e73bc6d245157b06f95e2a3ce30a3e6ed99cdd11088d: Status 404 returned error can't find the container with id aa31a4830233f3567bd5e73bc6d245157b06f95e2a3ce30a3e6ed99cdd11088d
Oct 30 20:41:48.312478 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:41:48.312445    4521 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e8b2d49_82b3_47fa_8317_a5df38e00cb4.slice/crio-c389e40b7d234b3d0ee586824e1da205d2e1503cfac70c3ca685b43ea42de7b3 WatchSource:0}: Error finding container c389e40b7d234b3d0ee586824e1da205d2e1503cfac70c3ca685b43ea42de7b3: Status 404 returned error can't find the container with id c389e40b7d234b3d0ee586824e1da205d2e1503cfac70c3ca685b43ea42de7b3
Oct 30 20:41:48.447735 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:48.447697    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644txvp8" event={"ID":"be269a25-d606-4ec1-acb4-38eedf4f201c","Type":"ContainerStarted","Data":"aa31a4830233f3567bd5e73bc6d245157b06f95e2a3ce30a3e6ed99cdd11088d"}
Oct 30 20:41:48.448724 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:48.448701    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c984t8" event={"ID":"1e8b2d49-82b3-47fa-8317-a5df38e00cb4","Type":"ContainerStarted","Data":"c389e40b7d234b3d0ee586824e1da205d2e1503cfac70c3ca685b43ea42de7b3"}
Oct 30 20:41:51.364904 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:51.364866    4521 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qtr6p"]
Oct 30 20:41:51.367775 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:51.367372    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qtr6p"
Oct 30 20:41:51.372426 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:51.372397    4521 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-8s6jn\""
Oct 30 20:41:51.377962 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:51.377595    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qtr6p"]
Oct 30 20:41:51.515848 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:51.515610    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvnlk\" (UniqueName: \"kubernetes.io/projected/de36102c-fee1-40dd-ba3f-95691343bae0-kube-api-access-pvnlk\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qtr6p\" (UID: \"de36102c-fee1-40dd-ba3f-95691343bae0\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qtr6p"
Oct 30 20:41:51.515848 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:51.515672    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/de36102c-fee1-40dd-ba3f-95691343bae0-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qtr6p\" (UID: \"de36102c-fee1-40dd-ba3f-95691343bae0\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qtr6p"
Oct 30 20:41:51.515848 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:51.515724    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/de36102c-fee1-40dd-ba3f-95691343bae0-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qtr6p\" (UID: \"de36102c-fee1-40dd-ba3f-95691343bae0\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qtr6p"
Oct 30 20:41:51.618106 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:51.616872    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/de36102c-fee1-40dd-ba3f-95691343bae0-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qtr6p\" (UID: \"de36102c-fee1-40dd-ba3f-95691343bae0\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qtr6p"
Oct 30 20:41:51.618106 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:51.616956    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvnlk\" (UniqueName: \"kubernetes.io/projected/de36102c-fee1-40dd-ba3f-95691343bae0-kube-api-access-pvnlk\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qtr6p\" (UID: \"de36102c-fee1-40dd-ba3f-95691343bae0\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qtr6p"
Oct 30 20:41:51.618106 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:51.616996    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/de36102c-fee1-40dd-ba3f-95691343bae0-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qtr6p\" (UID: \"de36102c-fee1-40dd-ba3f-95691343bae0\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qtr6p"
Oct 30 20:41:51.618106 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:51.617451    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/de36102c-fee1-40dd-ba3f-95691343bae0-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qtr6p\" (UID: \"de36102c-fee1-40dd-ba3f-95691343bae0\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qtr6p"
Oct 30 20:41:51.618106 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:51.617714    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/de36102c-fee1-40dd-ba3f-95691343bae0-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qtr6p\" (UID: \"de36102c-fee1-40dd-ba3f-95691343bae0\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qtr6p"
Oct 30 20:41:51.632291 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:51.632258    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvnlk\" (UniqueName: \"kubernetes.io/projected/de36102c-fee1-40dd-ba3f-95691343bae0-kube-api-access-pvnlk\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qtr6p\" (UID: \"de36102c-fee1-40dd-ba3f-95691343bae0\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qtr6p"
Oct 30 20:41:51.683234 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:51.683195    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qtr6p"
Oct 30 20:41:51.970850 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:51.970819    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qtr6p"]
Oct 30 20:41:51.974698 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:41:51.974664    4521 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde36102c_fee1_40dd_ba3f_95691343bae0.slice/crio-7aa97f4876e218bce99399f7e39f17fd5bd6fdf4fd5e0cc99b27dde58286a3a6 WatchSource:0}: Error finding container 7aa97f4876e218bce99399f7e39f17fd5bd6fdf4fd5e0cc99b27dde58286a3a6: Status 404 returned error can't find the container with id 7aa97f4876e218bce99399f7e39f17fd5bd6fdf4fd5e0cc99b27dde58286a3a6
Oct 30 20:41:52.466206 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:52.466165    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qtr6p" event={"ID":"de36102c-fee1-40dd-ba3f-95691343bae0","Type":"ContainerStarted","Data":"7aa97f4876e218bce99399f7e39f17fd5bd6fdf4fd5e0cc99b27dde58286a3a6"}
Oct 30 20:41:52.775905 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:52.775822    4521 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7txbfc"]
Oct 30 20:41:52.778615 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:52.778590    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7txbfc"
Oct 30 20:41:52.783671 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:52.783648    4521 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-lppnh\""
Oct 30 20:41:52.796985 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:52.796954    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7txbfc"]
Oct 30 20:41:52.873045 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:52.871868    4521 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak7782"]
Oct 30 20:41:52.875218 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:52.875187    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak7782"
Oct 30 20:41:52.882287 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:52.882232    4521 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-6hfp5\""
Oct 30 20:41:52.886894 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:52.886870    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak7782"]
Oct 30 20:41:52.928706 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:52.928667    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/824a2b93-8ec6-4fe1-9d0d-abf8cc679abf-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7txbfc\" (UID: \"824a2b93-8ec6-4fe1-9d0d-abf8cc679abf\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7txbfc"
Oct 30 20:41:52.928872 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:52.928732    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/62d1efb9-0cea-45ca-a468-31f81124bf4b-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak7782\" (UID: \"62d1efb9-0cea-45ca-a468-31f81124bf4b\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak7782"
Oct 30 20:41:52.928872 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:52.928764    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mq647\" (UniqueName: \"kubernetes.io/projected/62d1efb9-0cea-45ca-a468-31f81124bf4b-kube-api-access-mq647\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak7782\" (UID: \"62d1efb9-0cea-45ca-a468-31f81124bf4b\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak7782"
Oct 30 20:41:52.928872 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:52.928794    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfm7n\" (UniqueName: \"kubernetes.io/projected/824a2b93-8ec6-4fe1-9d0d-abf8cc679abf-kube-api-access-zfm7n\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7txbfc\" (UID: \"824a2b93-8ec6-4fe1-9d0d-abf8cc679abf\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7txbfc"
Oct 30 20:41:52.929018 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:52.928875    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/824a2b93-8ec6-4fe1-9d0d-abf8cc679abf-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7txbfc\" (UID: \"824a2b93-8ec6-4fe1-9d0d-abf8cc679abf\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7txbfc"
Oct 30 20:41:52.929018 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:52.928949    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/62d1efb9-0cea-45ca-a468-31f81124bf4b-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak7782\" (UID: \"62d1efb9-0cea-45ca-a468-31f81124bf4b\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak7782"
Oct 30 20:41:52.996051 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:52.996013    4521 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cpdbc"]
Oct 30 20:41:52.998595 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:52.998572    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cpdbc"
Oct 30 20:41:53.003192 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:53.003166    4521 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-pkl6c\""
Oct 30 20:41:53.009946 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:53.009901    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cpdbc"]
Oct 30 20:41:53.030734 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:53.029550    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/62d1efb9-0cea-45ca-a468-31f81124bf4b-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak7782\" (UID: \"62d1efb9-0cea-45ca-a468-31f81124bf4b\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak7782"
Oct 30 20:41:53.030734 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:53.029600    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mq647\" (UniqueName: \"kubernetes.io/projected/62d1efb9-0cea-45ca-a468-31f81124bf4b-kube-api-access-mq647\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak7782\" (UID: \"62d1efb9-0cea-45ca-a468-31f81124bf4b\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak7782"
Oct 30 20:41:53.030734 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:53.029634    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfm7n\" (UniqueName: \"kubernetes.io/projected/824a2b93-8ec6-4fe1-9d0d-abf8cc679abf-kube-api-access-zfm7n\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7txbfc\" (UID: \"824a2b93-8ec6-4fe1-9d0d-abf8cc679abf\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7txbfc"
Oct 30 20:41:53.030734 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:53.029673    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/824a2b93-8ec6-4fe1-9d0d-abf8cc679abf-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7txbfc\" (UID: \"824a2b93-8ec6-4fe1-9d0d-abf8cc679abf\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7txbfc"
Oct 30 20:41:53.030734 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:53.029715    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/62d1efb9-0cea-45ca-a468-31f81124bf4b-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak7782\" (UID: \"62d1efb9-0cea-45ca-a468-31f81124bf4b\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak7782"
Oct 30 20:41:53.030734 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:53.029760    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/824a2b93-8ec6-4fe1-9d0d-abf8cc679abf-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7txbfc\" (UID: \"824a2b93-8ec6-4fe1-9d0d-abf8cc679abf\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7txbfc"
Oct 30 20:41:53.030734 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:53.030022    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/62d1efb9-0cea-45ca-a468-31f81124bf4b-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak7782\" (UID: \"62d1efb9-0cea-45ca-a468-31f81124bf4b\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak7782"
Oct 30 20:41:53.030734 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:53.030163    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/824a2b93-8ec6-4fe1-9d0d-abf8cc679abf-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7txbfc\" (UID: \"824a2b93-8ec6-4fe1-9d0d-abf8cc679abf\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7txbfc"
Oct 30 20:41:53.030734 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:53.030432    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/824a2b93-8ec6-4fe1-9d0d-abf8cc679abf-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7txbfc\" (UID: \"824a2b93-8ec6-4fe1-9d0d-abf8cc679abf\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7txbfc"
Oct 30 20:41:53.030734 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:53.030658    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/62d1efb9-0cea-45ca-a468-31f81124bf4b-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak7782\" (UID: \"62d1efb9-0cea-45ca-a468-31f81124bf4b\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak7782"
Oct 30 20:41:53.045912 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:53.045882    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mq647\" (UniqueName: \"kubernetes.io/projected/62d1efb9-0cea-45ca-a468-31f81124bf4b-kube-api-access-mq647\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak7782\" (UID: \"62d1efb9-0cea-45ca-a468-31f81124bf4b\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak7782"
Oct 30 20:41:53.048238 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:53.048213    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfm7n\" (UniqueName: \"kubernetes.io/projected/824a2b93-8ec6-4fe1-9d0d-abf8cc679abf-kube-api-access-zfm7n\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7txbfc\" (UID: \"824a2b93-8ec6-4fe1-9d0d-abf8cc679abf\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7txbfc"
Oct 30 20:41:53.070122 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:53.070095    4521 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cd4bdk"]
Oct 30 20:41:53.072803 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:53.072781    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cd4bdk"
Oct 30 20:41:53.088816 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:53.088784    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cd4bdk"]
Oct 30 20:41:53.102397 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:53.102369    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7txbfc"
Oct 30 20:41:53.103830 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:53.103807    4521 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-7ljxc\""
Oct 30 20:41:53.130931 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:53.130899    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9dtj\" (UniqueName: \"kubernetes.io/projected/3963bd73-a6f8-4650-a9ef-8ffd1a2ee63f-kube-api-access-l9dtj\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cpdbc\" (UID: \"3963bd73-a6f8-4650-a9ef-8ffd1a2ee63f\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cpdbc"
Oct 30 20:41:53.131106 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:53.130979    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/54d00e9e-313d-4fde-87a2-e641c4caf568-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cd4bdk\" (UID: \"54d00e9e-313d-4fde-87a2-e641c4caf568\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cd4bdk"
Oct 30 20:41:53.131106 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:53.131012    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/54d00e9e-313d-4fde-87a2-e641c4caf568-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cd4bdk\" (UID: \"54d00e9e-313d-4fde-87a2-e641c4caf568\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cd4bdk"
Oct 30 20:41:53.131106 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:53.131043    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnhh8\" (UniqueName: \"kubernetes.io/projected/54d00e9e-313d-4fde-87a2-e641c4caf568-kube-api-access-dnhh8\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cd4bdk\" (UID: \"54d00e9e-313d-4fde-87a2-e641c4caf568\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cd4bdk"
Oct 30 20:41:53.131106 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:53.131079    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3963bd73-a6f8-4650-a9ef-8ffd1a2ee63f-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cpdbc\" (UID: \"3963bd73-a6f8-4650-a9ef-8ffd1a2ee63f\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cpdbc"
Oct 30 20:41:53.131299 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:53.131112    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3963bd73-a6f8-4650-a9ef-8ffd1a2ee63f-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cpdbc\" (UID: \"3963bd73-a6f8-4650-a9ef-8ffd1a2ee63f\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cpdbc"
Oct 30 20:41:53.199461 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:53.199181    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak7782"
Oct 30 20:41:53.231734 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:53.231692    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/54d00e9e-313d-4fde-87a2-e641c4caf568-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cd4bdk\" (UID: \"54d00e9e-313d-4fde-87a2-e641c4caf568\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cd4bdk"
Oct 30 20:41:53.231903 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:53.231750    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/54d00e9e-313d-4fde-87a2-e641c4caf568-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cd4bdk\" (UID: \"54d00e9e-313d-4fde-87a2-e641c4caf568\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cd4bdk"
Oct 30 20:41:53.231903 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:53.231791    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnhh8\" (UniqueName: \"kubernetes.io/projected/54d00e9e-313d-4fde-87a2-e641c4caf568-kube-api-access-dnhh8\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cd4bdk\" (UID: \"54d00e9e-313d-4fde-87a2-e641c4caf568\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cd4bdk"
Oct 30 20:41:53.231903 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:53.231830    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3963bd73-a6f8-4650-a9ef-8ffd1a2ee63f-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cpdbc\" (UID: \"3963bd73-a6f8-4650-a9ef-8ffd1a2ee63f\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cpdbc"
Oct 30 20:41:53.231903 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:53.231870    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3963bd73-a6f8-4650-a9ef-8ffd1a2ee63f-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cpdbc\" (UID: \"3963bd73-a6f8-4650-a9ef-8ffd1a2ee63f\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cpdbc"
Oct 30 20:41:53.232074 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:53.231984    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9dtj\" (UniqueName: \"kubernetes.io/projected/3963bd73-a6f8-4650-a9ef-8ffd1a2ee63f-kube-api-access-l9dtj\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cpdbc\" (UID: \"3963bd73-a6f8-4650-a9ef-8ffd1a2ee63f\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cpdbc"
Oct 30 20:41:53.232173 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:53.232150    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/54d00e9e-313d-4fde-87a2-e641c4caf568-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cd4bdk\" (UID: \"54d00e9e-313d-4fde-87a2-e641c4caf568\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cd4bdk"
Oct 30 20:41:53.232679 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:53.232648    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/54d00e9e-313d-4fde-87a2-e641c4caf568-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cd4bdk\" (UID: \"54d00e9e-313d-4fde-87a2-e641c4caf568\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cd4bdk"
Oct 30 20:41:53.232847 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:53.232821    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3963bd73-a6f8-4650-a9ef-8ffd1a2ee63f-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cpdbc\" (UID: \"3963bd73-a6f8-4650-a9ef-8ffd1a2ee63f\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cpdbc"
Oct 30 20:41:53.232940 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:53.232920    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3963bd73-a6f8-4650-a9ef-8ffd1a2ee63f-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cpdbc\" (UID: \"3963bd73-a6f8-4650-a9ef-8ffd1a2ee63f\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cpdbc"
Oct 30 20:41:53.245314 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:53.245266    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9dtj\" (UniqueName: \"kubernetes.io/projected/3963bd73-a6f8-4650-a9ef-8ffd1a2ee63f-kube-api-access-l9dtj\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cpdbc\" (UID: \"3963bd73-a6f8-4650-a9ef-8ffd1a2ee63f\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cpdbc"
Oct 30 20:41:53.245314 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:53.245299    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnhh8\" (UniqueName: \"kubernetes.io/projected/54d00e9e-313d-4fde-87a2-e641c4caf568-kube-api-access-dnhh8\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cd4bdk\" (UID: \"54d00e9e-313d-4fde-87a2-e641c4caf568\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cd4bdk"
Oct 30 20:41:53.317080 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:53.317003    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cpdbc"
Oct 30 20:41:53.399349 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:53.396668    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cd4bdk"
Oct 30 20:41:53.408152 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:53.408119    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7txbfc"]
Oct 30 20:41:53.477417 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:53.477336    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7txbfc" event={"ID":"824a2b93-8ec6-4fe1-9d0d-abf8cc679abf","Type":"ContainerStarted","Data":"2c2bbc95293c77b15523a1f3d6668e96c8b8f97a6ffbc20c09b7f5f49f952958"}
Oct 30 20:41:53.495365 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:53.495170    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak7782"]
Oct 30 20:41:53.498736 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:41:53.498708    4521 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62d1efb9_0cea_45ca_a468_31f81124bf4b.slice/crio-865a529ec5ebc9de8190e5c933563a4d8b6a1b77d3bf9fe2f7eb66621aadee07 WatchSource:0}: Error finding container 865a529ec5ebc9de8190e5c933563a4d8b6a1b77d3bf9fe2f7eb66621aadee07: Status 404 returned error can't find the container with id 865a529ec5ebc9de8190e5c933563a4d8b6a1b77d3bf9fe2f7eb66621aadee07
Oct 30 20:41:53.618663 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:53.615726    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cpdbc"]
Oct 30 20:41:53.618663 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:53.617090    4521 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-bckjb"]
Oct 30 20:41:53.620093 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:53.620074    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-bckjb"
Oct 30 20:41:53.625381 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:53.625358    4521 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 30 20:41:53.646022 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:53.645991    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-bckjb"]
Oct 30 20:41:53.662227 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:53.662166    4521 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-6qg5v\""
Oct 30 20:41:53.689060 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:53.689014    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cd4bdk"]
Oct 30 20:41:53.693606 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:41:53.693579    4521 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54d00e9e_313d_4fde_87a2_e641c4caf568.slice/crio-b50ea07ab6529e58675b4b0fdef7dca5f2cbc900e7f6e20d788bd447b57b5369 WatchSource:0}: Error finding container b50ea07ab6529e58675b4b0fdef7dca5f2cbc900e7f6e20d788bd447b57b5369: Status 404 returned error can't find the container with id b50ea07ab6529e58675b4b0fdef7dca5f2cbc900e7f6e20d788bd447b57b5369
Oct 30 20:41:53.714779 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:53.714749    4521 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-bc8769f9-rxf86"]
Oct 30 20:41:53.717819 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:53.717441    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-bc8769f9-rxf86"
Oct 30 20:41:53.730499 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:53.725985    4521 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 30 20:41:53.730499 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:53.726081    4521 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 30 20:41:53.738714 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:53.738670    4521 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-f2bds\""
Oct 30 20:41:53.739641 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:53.738829    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62149acf-8605-4f8d-b1ac-6e8d6b4865fb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-bckjb\" (UID: \"62149acf-8605-4f8d-b1ac-6e8d6b4865fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-bckjb"
Oct 30 20:41:53.739641 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:53.738858    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62149acf-8605-4f8d-b1ac-6e8d6b4865fb-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-bckjb\" (UID: \"62149acf-8605-4f8d-b1ac-6e8d6b4865fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-bckjb"
Oct 30 20:41:53.739641 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:53.738879    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62149acf-8605-4f8d-b1ac-6e8d6b4865fb-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-bckjb\" (UID: \"62149acf-8605-4f8d-b1ac-6e8d6b4865fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-bckjb"
Oct 30 20:41:53.739641 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:53.738918    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62149acf-8605-4f8d-b1ac-6e8d6b4865fb-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-bckjb\" (UID: \"62149acf-8605-4f8d-b1ac-6e8d6b4865fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-bckjb"
Oct 30 20:41:53.739641 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:53.738939    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nf4vw\" (UniqueName: \"kubernetes.io/projected/62149acf-8605-4f8d-b1ac-6e8d6b4865fb-kube-api-access-nf4vw\") pod \"maas-default-gateway-openshift-default-58c6585447-bckjb\" (UID: \"62149acf-8605-4f8d-b1ac-6e8d6b4865fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-bckjb"
Oct 30 20:41:53.739641 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:53.738969    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62149acf-8605-4f8d-b1ac-6e8d6b4865fb-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-bckjb\" (UID: \"62149acf-8605-4f8d-b1ac-6e8d6b4865fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-bckjb"
Oct 30 20:41:53.739641 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:53.738992    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62149acf-8605-4f8d-b1ac-6e8d6b4865fb-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-bckjb\" (UID: \"62149acf-8605-4f8d-b1ac-6e8d6b4865fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-bckjb"
Oct 30 20:41:53.739641 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:53.739007    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62149acf-8605-4f8d-b1ac-6e8d6b4865fb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-bckjb\" (UID: \"62149acf-8605-4f8d-b1ac-6e8d6b4865fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-bckjb"
Oct 30 20:41:53.739641 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:53.739027    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62149acf-8605-4f8d-b1ac-6e8d6b4865fb-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-bckjb\" (UID: \"62149acf-8605-4f8d-b1ac-6e8d6b4865fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-bckjb"
Oct 30 20:41:53.741224 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:53.741196    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-bc8769f9-rxf86"]
Oct 30 20:41:53.840379 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:53.840346    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f421589f-1efe-4427-a723-435502b4a14b-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-bc8769f9-rxf86\" (UID: \"f421589f-1efe-4427-a723-435502b4a14b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-bc8769f9-rxf86"
Oct 30 20:41:53.840379 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:53.840390    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f421589f-1efe-4427-a723-435502b4a14b-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-bc8769f9-rxf86\" (UID: \"f421589f-1efe-4427-a723-435502b4a14b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-bc8769f9-rxf86"
Oct 30 20:41:53.840604 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:53.840427    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62149acf-8605-4f8d-b1ac-6e8d6b4865fb-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-bckjb\" (UID: \"62149acf-8605-4f8d-b1ac-6e8d6b4865fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-bckjb"
Oct 30 20:41:53.840604 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:53.840459    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62149acf-8605-4f8d-b1ac-6e8d6b4865fb-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-bckjb\" (UID: \"62149acf-8605-4f8d-b1ac-6e8d6b4865fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-bckjb"
Oct 30 20:41:53.840604 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:53.840482    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62149acf-8605-4f8d-b1ac-6e8d6b4865fb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-bckjb\" (UID: \"62149acf-8605-4f8d-b1ac-6e8d6b4865fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-bckjb"
Oct 30 20:41:53.840604 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:53.840517    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62149acf-8605-4f8d-b1ac-6e8d6b4865fb-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-bckjb\" (UID: \"62149acf-8605-4f8d-b1ac-6e8d6b4865fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-bckjb"
Oct 30 20:41:53.840604 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:53.840540    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f421589f-1efe-4427-a723-435502b4a14b-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-bc8769f9-rxf86\" (UID: \"f421589f-1efe-4427-a723-435502b4a14b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-bc8769f9-rxf86"
Oct 30 20:41:53.840604 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:53.840571    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdqkp\" (UniqueName: \"kubernetes.io/projected/f421589f-1efe-4427-a723-435502b4a14b-kube-api-access-tdqkp\") pod \"openshift-ai-inference-openshift-ai-inference-bc8769f9-rxf86\" (UID: \"f421589f-1efe-4427-a723-435502b4a14b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-bc8769f9-rxf86"
Oct 30 20:41:53.840913 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:53.840606    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f421589f-1efe-4427-a723-435502b4a14b-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-bc8769f9-rxf86\" (UID: \"f421589f-1efe-4427-a723-435502b4a14b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-bc8769f9-rxf86"
Oct 30 20:41:53.840913 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:53.840641    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f421589f-1efe-4427-a723-435502b4a14b-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-bc8769f9-rxf86\" (UID: \"f421589f-1efe-4427-a723-435502b4a14b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-bc8769f9-rxf86"
Oct 30 20:41:53.840913 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:53.840673    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62149acf-8605-4f8d-b1ac-6e8d6b4865fb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-bckjb\" (UID: \"62149acf-8605-4f8d-b1ac-6e8d6b4865fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-bckjb"
Oct 30 20:41:53.840913 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:53.840707    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f421589f-1efe-4427-a723-435502b4a14b-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-bc8769f9-rxf86\" (UID: \"f421589f-1efe-4427-a723-435502b4a14b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-bc8769f9-rxf86"
Oct 30 20:41:53.840913 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:53.840739    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62149acf-8605-4f8d-b1ac-6e8d6b4865fb-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-bckjb\" (UID: \"62149acf-8605-4f8d-b1ac-6e8d6b4865fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-bckjb"
Oct 30 20:41:53.840913 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:53.840772    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62149acf-8605-4f8d-b1ac-6e8d6b4865fb-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-bckjb\" (UID: \"62149acf-8605-4f8d-b1ac-6e8d6b4865fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-bckjb"
Oct 30 20:41:53.840913 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:53.840835    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f421589f-1efe-4427-a723-435502b4a14b-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-bc8769f9-rxf86\" (UID: \"f421589f-1efe-4427-a723-435502b4a14b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-bc8769f9-rxf86"
Oct 30 20:41:53.840913 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:53.840863    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62149acf-8605-4f8d-b1ac-6e8d6b4865fb-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-bckjb\" (UID: \"62149acf-8605-4f8d-b1ac-6e8d6b4865fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-bckjb"
Oct 30 20:41:53.840913 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:53.840888    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nf4vw\" (UniqueName: \"kubernetes.io/projected/62149acf-8605-4f8d-b1ac-6e8d6b4865fb-kube-api-access-nf4vw\") pod \"maas-default-gateway-openshift-default-58c6585447-bckjb\" (UID: \"62149acf-8605-4f8d-b1ac-6e8d6b4865fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-bckjb"
Oct 30 20:41:53.841350 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:53.840926    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f421589f-1efe-4427-a723-435502b4a14b-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-bc8769f9-rxf86\" (UID: \"f421589f-1efe-4427-a723-435502b4a14b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-bc8769f9-rxf86"
Oct 30 20:41:53.841461 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:53.841439    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62149acf-8605-4f8d-b1ac-6e8d6b4865fb-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-bckjb\" (UID: \"62149acf-8605-4f8d-b1ac-6e8d6b4865fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-bckjb"
Oct 30 20:41:53.849163 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:53.842286    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62149acf-8605-4f8d-b1ac-6e8d6b4865fb-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-bckjb\" (UID: \"62149acf-8605-4f8d-b1ac-6e8d6b4865fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-bckjb"
Oct 30 20:41:53.849163 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:53.841659    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62149acf-8605-4f8d-b1ac-6e8d6b4865fb-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-bckjb\" (UID: \"62149acf-8605-4f8d-b1ac-6e8d6b4865fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-bckjb"
Oct 30 20:41:53.849163 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:53.843144    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62149acf-8605-4f8d-b1ac-6e8d6b4865fb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-bckjb\" (UID: \"62149acf-8605-4f8d-b1ac-6e8d6b4865fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-bckjb"
Oct 30 20:41:53.849163 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:53.843376    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62149acf-8605-4f8d-b1ac-6e8d6b4865fb-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-bckjb\" (UID: \"62149acf-8605-4f8d-b1ac-6e8d6b4865fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-bckjb"
Oct 30 20:41:53.849163 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:53.845000    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62149acf-8605-4f8d-b1ac-6e8d6b4865fb-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-bckjb\" (UID: \"62149acf-8605-4f8d-b1ac-6e8d6b4865fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-bckjb"
Oct 30 20:41:53.849676 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:53.849654    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62149acf-8605-4f8d-b1ac-6e8d6b4865fb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-bckjb\" (UID: \"62149acf-8605-4f8d-b1ac-6e8d6b4865fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-bckjb"
Oct 30 20:41:53.856709 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:53.856683    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62149acf-8605-4f8d-b1ac-6e8d6b4865fb-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-bckjb\" (UID: \"62149acf-8605-4f8d-b1ac-6e8d6b4865fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-bckjb"
Oct 30 20:41:53.859656 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:53.859610    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nf4vw\" (UniqueName: \"kubernetes.io/projected/62149acf-8605-4f8d-b1ac-6e8d6b4865fb-kube-api-access-nf4vw\") pod \"maas-default-gateway-openshift-default-58c6585447-bckjb\" (UID: \"62149acf-8605-4f8d-b1ac-6e8d6b4865fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-bckjb"
Oct 30 20:41:53.939694 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:53.939372    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-bckjb"
Oct 30 20:41:53.943371 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:53.941387    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f421589f-1efe-4427-a723-435502b4a14b-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-bc8769f9-rxf86\" (UID: \"f421589f-1efe-4427-a723-435502b4a14b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-bc8769f9-rxf86"
Oct 30 20:41:53.943371 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:53.941426    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f421589f-1efe-4427-a723-435502b4a14b-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-bc8769f9-rxf86\" (UID: \"f421589f-1efe-4427-a723-435502b4a14b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-bc8769f9-rxf86"
Oct 30 20:41:53.943371 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:53.941486    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f421589f-1efe-4427-a723-435502b4a14b-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-bc8769f9-rxf86\" (UID: \"f421589f-1efe-4427-a723-435502b4a14b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-bc8769f9-rxf86"
Oct 30 20:41:53.943371 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:53.941520    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f421589f-1efe-4427-a723-435502b4a14b-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-bc8769f9-rxf86\" (UID: \"f421589f-1efe-4427-a723-435502b4a14b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-bc8769f9-rxf86"
Oct 30 20:41:53.943371 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:53.941550    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f421589f-1efe-4427-a723-435502b4a14b-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-bc8769f9-rxf86\" (UID: \"f421589f-1efe-4427-a723-435502b4a14b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-bc8769f9-rxf86"
Oct 30 20:41:53.943371 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:53.941576    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f421589f-1efe-4427-a723-435502b4a14b-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-bc8769f9-rxf86\" (UID: \"f421589f-1efe-4427-a723-435502b4a14b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-bc8769f9-rxf86"
Oct 30 20:41:53.943371 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:53.941630    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f421589f-1efe-4427-a723-435502b4a14b-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-bc8769f9-rxf86\" (UID: \"f421589f-1efe-4427-a723-435502b4a14b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-bc8769f9-rxf86"
Oct 30 20:41:53.943371 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:53.941661    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdqkp\" (UniqueName: \"kubernetes.io/projected/f421589f-1efe-4427-a723-435502b4a14b-kube-api-access-tdqkp\") pod \"openshift-ai-inference-openshift-ai-inference-bc8769f9-rxf86\" (UID: \"f421589f-1efe-4427-a723-435502b4a14b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-bc8769f9-rxf86"
Oct 30 20:41:53.943371 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:53.941753    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f421589f-1efe-4427-a723-435502b4a14b-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-bc8769f9-rxf86\" (UID: \"f421589f-1efe-4427-a723-435502b4a14b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-bc8769f9-rxf86"
Oct 30 20:41:53.943371 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:53.942235    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f421589f-1efe-4427-a723-435502b4a14b-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-bc8769f9-rxf86\" (UID: \"f421589f-1efe-4427-a723-435502b4a14b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-bc8769f9-rxf86"
Oct 30 20:41:53.943371 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:53.942828    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f421589f-1efe-4427-a723-435502b4a14b-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-bc8769f9-rxf86\" (UID: \"f421589f-1efe-4427-a723-435502b4a14b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-bc8769f9-rxf86"
Oct 30 20:41:53.945227 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:53.944971    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f421589f-1efe-4427-a723-435502b4a14b-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-bc8769f9-rxf86\" (UID: \"f421589f-1efe-4427-a723-435502b4a14b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-bc8769f9-rxf86"
Oct 30 20:41:53.945227 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:53.945170    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f421589f-1efe-4427-a723-435502b4a14b-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-bc8769f9-rxf86\" (UID: \"f421589f-1efe-4427-a723-435502b4a14b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-bc8769f9-rxf86"
Oct 30 20:41:53.945227 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:53.945197    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f421589f-1efe-4427-a723-435502b4a14b-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-bc8769f9-rxf86\" (UID: \"f421589f-1efe-4427-a723-435502b4a14b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-bc8769f9-rxf86"
Oct 30 20:41:53.947358 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:53.947288    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f421589f-1efe-4427-a723-435502b4a14b-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-bc8769f9-rxf86\" (UID: \"f421589f-1efe-4427-a723-435502b4a14b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-bc8769f9-rxf86"
Oct 30 20:41:53.951658 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:53.951612    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f421589f-1efe-4427-a723-435502b4a14b-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-bc8769f9-rxf86\" (UID: \"f421589f-1efe-4427-a723-435502b4a14b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-bc8769f9-rxf86"
Oct 30 20:41:53.959651 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:53.959599    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f421589f-1efe-4427-a723-435502b4a14b-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-bc8769f9-rxf86\" (UID: \"f421589f-1efe-4427-a723-435502b4a14b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-bc8769f9-rxf86"
Oct 30 20:41:53.959920 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:53.959897    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdqkp\" (UniqueName: \"kubernetes.io/projected/f421589f-1efe-4427-a723-435502b4a14b-kube-api-access-tdqkp\") pod \"openshift-ai-inference-openshift-ai-inference-bc8769f9-rxf86\" (UID: \"f421589f-1efe-4427-a723-435502b4a14b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-bc8769f9-rxf86"
Oct 30 20:41:54.034116 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:54.033671    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-bc8769f9-rxf86"
Oct 30 20:41:54.233831 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:54.233751    4521 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccd69g"]
Oct 30 20:41:54.236299 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:54.236270    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccd69g"
Oct 30 20:41:54.249193 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:54.249161    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccd69g"]
Oct 30 20:41:54.262207 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:54.262094    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-bckjb"]
Oct 30 20:41:54.346298 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:54.346057    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fd75cc69-c4d2-4e9f-9215-f0e0b81bd64e-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccd69g\" (UID: \"fd75cc69-c4d2-4e9f-9215-f0e0b81bd64e\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccd69g"
Oct 30 20:41:54.346298 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:54.346126    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7m2mf\" (UniqueName: \"kubernetes.io/projected/fd75cc69-c4d2-4e9f-9215-f0e0b81bd64e-kube-api-access-7m2mf\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccd69g\" (UID: \"fd75cc69-c4d2-4e9f-9215-f0e0b81bd64e\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccd69g"
Oct 30 20:41:54.346298 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:54.346176    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fd75cc69-c4d2-4e9f-9215-f0e0b81bd64e-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccd69g\" (UID: \"fd75cc69-c4d2-4e9f-9215-f0e0b81bd64e\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccd69g"
Oct 30 20:41:54.369441 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:54.369384    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-bc8769f9-rxf86"]
Oct 30 20:41:54.378840 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:41:54.378805    4521 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf421589f_1efe_4427_a723_435502b4a14b.slice/crio-258f3181f7a312f2170a85b8d97ec4bce19672d9d793ea150a9e6fecb40e6740 WatchSource:0}: Error finding container 258f3181f7a312f2170a85b8d97ec4bce19672d9d793ea150a9e6fecb40e6740: Status 404 returned error can't find the container with id 258f3181f7a312f2170a85b8d97ec4bce19672d9d793ea150a9e6fecb40e6740
Oct 30 20:41:54.447371 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:54.447316    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fd75cc69-c4d2-4e9f-9215-f0e0b81bd64e-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccd69g\" (UID: \"fd75cc69-c4d2-4e9f-9215-f0e0b81bd64e\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccd69g"
Oct 30 20:41:54.447566 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:54.447435    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fd75cc69-c4d2-4e9f-9215-f0e0b81bd64e-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccd69g\" (UID: \"fd75cc69-c4d2-4e9f-9215-f0e0b81bd64e\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccd69g"
Oct 30 20:41:54.447566 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:54.447490    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7m2mf\" (UniqueName: \"kubernetes.io/projected/fd75cc69-c4d2-4e9f-9215-f0e0b81bd64e-kube-api-access-7m2mf\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccd69g\" (UID: \"fd75cc69-c4d2-4e9f-9215-f0e0b81bd64e\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccd69g"
Oct 30 20:41:54.448431 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:54.448217    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fd75cc69-c4d2-4e9f-9215-f0e0b81bd64e-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccd69g\" (UID: \"fd75cc69-c4d2-4e9f-9215-f0e0b81bd64e\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccd69g"
Oct 30 20:41:54.448523 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:54.448494    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fd75cc69-c4d2-4e9f-9215-f0e0b81bd64e-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccd69g\" (UID: \"fd75cc69-c4d2-4e9f-9215-f0e0b81bd64e\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccd69g"
Oct 30 20:41:54.462930 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:54.462896    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7m2mf\" (UniqueName: \"kubernetes.io/projected/fd75cc69-c4d2-4e9f-9215-f0e0b81bd64e-kube-api-access-7m2mf\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccd69g\" (UID: \"fd75cc69-c4d2-4e9f-9215-f0e0b81bd64e\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccd69g"
Oct 30 20:41:54.484531 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:54.484396    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak7782" event={"ID":"62d1efb9-0cea-45ca-a468-31f81124bf4b","Type":"ContainerStarted","Data":"865a529ec5ebc9de8190e5c933563a4d8b6a1b77d3bf9fe2f7eb66621aadee07"}
Oct 30 20:41:54.486402 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:54.486308    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-bc8769f9-rxf86" event={"ID":"f421589f-1efe-4427-a723-435502b4a14b","Type":"ContainerStarted","Data":"258f3181f7a312f2170a85b8d97ec4bce19672d9d793ea150a9e6fecb40e6740"}
Oct 30 20:41:54.488987 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:54.488912    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cd4bdk" event={"ID":"54d00e9e-313d-4fde-87a2-e641c4caf568","Type":"ContainerStarted","Data":"b50ea07ab6529e58675b4b0fdef7dca5f2cbc900e7f6e20d788bd447b57b5369"}
Oct 30 20:41:54.492764 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:54.492685    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-bckjb" event={"ID":"62149acf-8605-4f8d-b1ac-6e8d6b4865fb","Type":"ContainerStarted","Data":"6f3be0af6d65d91a0537b8770d12f720a1c49714c00964b70c9752f042cba6bb"}
Oct 30 20:41:54.494777 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:54.494715    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cpdbc" event={"ID":"3963bd73-a6f8-4650-a9ef-8ffd1a2ee63f","Type":"ContainerStarted","Data":"0af16f3d3feadda1dea1e39cf3ec0ff87a01ed1869bf623c377d60a8dd6bffee"}
Oct 30 20:41:54.557983 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:54.557757    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccd69g"
Oct 30 20:41:54.947156 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:54.947098    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccd69g"]
Oct 30 20:41:54.951879 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:41:54.951849    4521 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd75cc69_c4d2_4e9f_9215_f0e0b81bd64e.slice/crio-e5fb688f9cf8a50423bc911570d09a09f00f3b515c5521c303fe33c1d06eca3c WatchSource:0}: Error finding container e5fb688f9cf8a50423bc911570d09a09f00f3b515c5521c303fe33c1d06eca3c: Status 404 returned error can't find the container with id e5fb688f9cf8a50423bc911570d09a09f00f3b515c5521c303fe33c1d06eca3c
Oct 30 20:41:54.974014 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:54.972374    4521 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644l85qk"]
Oct 30 20:41:54.975769 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:54.975745    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644l85qk"
Oct 30 20:41:54.991283 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:54.991216    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644l85qk"]
Oct 30 20:41:55.054241 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:55.053959    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5nvh\" (UniqueName: \"kubernetes.io/projected/747d9d7a-fdca-4839-afe2-ef5933b3cc16-kube-api-access-h5nvh\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644l85qk\" (UID: \"747d9d7a-fdca-4839-afe2-ef5933b3cc16\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644l85qk"
Oct 30 20:41:55.054241 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:55.054015    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/747d9d7a-fdca-4839-afe2-ef5933b3cc16-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644l85qk\" (UID: \"747d9d7a-fdca-4839-afe2-ef5933b3cc16\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644l85qk"
Oct 30 20:41:55.054241 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:55.054042    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/747d9d7a-fdca-4839-afe2-ef5933b3cc16-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644l85qk\" (UID: \"747d9d7a-fdca-4839-afe2-ef5933b3cc16\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644l85qk"
Oct 30 20:41:55.156318 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:55.155094    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5nvh\" (UniqueName: \"kubernetes.io/projected/747d9d7a-fdca-4839-afe2-ef5933b3cc16-kube-api-access-h5nvh\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644l85qk\" (UID: \"747d9d7a-fdca-4839-afe2-ef5933b3cc16\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644l85qk"
Oct 30 20:41:55.156318 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:55.155151    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/747d9d7a-fdca-4839-afe2-ef5933b3cc16-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644l85qk\" (UID: \"747d9d7a-fdca-4839-afe2-ef5933b3cc16\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644l85qk"
Oct 30 20:41:55.156318 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:55.155177    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/747d9d7a-fdca-4839-afe2-ef5933b3cc16-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644l85qk\" (UID: \"747d9d7a-fdca-4839-afe2-ef5933b3cc16\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644l85qk"
Oct 30 20:41:55.156318 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:55.155689    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/747d9d7a-fdca-4839-afe2-ef5933b3cc16-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644l85qk\" (UID: \"747d9d7a-fdca-4839-afe2-ef5933b3cc16\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644l85qk"
Oct 30 20:41:55.156318 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:55.156274    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/747d9d7a-fdca-4839-afe2-ef5933b3cc16-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644l85qk\" (UID: \"747d9d7a-fdca-4839-afe2-ef5933b3cc16\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644l85qk"
Oct 30 20:41:55.179258 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:55.179160    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5nvh\" (UniqueName: \"kubernetes.io/projected/747d9d7a-fdca-4839-afe2-ef5933b3cc16-kube-api-access-h5nvh\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644l85qk\" (UID: \"747d9d7a-fdca-4839-afe2-ef5933b3cc16\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644l85qk"
Oct 30 20:41:55.298304 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:55.297990    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644l85qk"
Oct 30 20:41:55.503408 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:55.503310    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccd69g" event={"ID":"fd75cc69-c4d2-4e9f-9215-f0e0b81bd64e","Type":"ContainerStarted","Data":"e5fb688f9cf8a50423bc911570d09a09f00f3b515c5521c303fe33c1d06eca3c"}
Oct 30 20:41:55.640866 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:55.640776    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644l85qk"]
Oct 30 20:41:56.546749 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:56.546666    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644l85qk" event={"ID":"747d9d7a-fdca-4839-afe2-ef5933b3cc16","Type":"ContainerStarted","Data":"d98d59991570206c9ef411cfedb636efc1e8b458698e5a5784c767245ce06280"}
Oct 30 20:41:58.565670 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:58.565624    4521 generic.go:356] "Generic (PLEG): container finished" podID="1e8b2d49-82b3-47fa-8317-a5df38e00cb4" containerID="b08b5d603a37e356583d06cfd0c1d2bceaa38f982954fa3accd3b26cbb19ffa5" exitCode=0
Oct 30 20:41:58.566798 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:58.566435    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c984t8" event={"ID":"1e8b2d49-82b3-47fa-8317-a5df38e00cb4","Type":"ContainerDied","Data":"b08b5d603a37e356583d06cfd0c1d2bceaa38f982954fa3accd3b26cbb19ffa5"}
Oct 30 20:41:58.573303 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:58.573157    4521 generic.go:356] "Generic (PLEG): container finished" podID="be269a25-d606-4ec1-acb4-38eedf4f201c" containerID="2b6121c16f9a9448ec65debf5dff8d617e31ac55ad9eaa68fa2170008df8083a" exitCode=0
Oct 30 20:41:58.573303 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:41:58.573223    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644txvp8" event={"ID":"be269a25-d606-4ec1-acb4-38eedf4f201c","Type":"ContainerDied","Data":"2b6121c16f9a9448ec65debf5dff8d617e31ac55ad9eaa68fa2170008df8083a"}
Oct 30 20:42:02.624396 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:02.624354    4521 generic.go:356] "Generic (PLEG): container finished" podID="de36102c-fee1-40dd-ba3f-95691343bae0" containerID="8b33ae365c9472b2cd3f5def3d436b433d65027287cba5583046f76423879257" exitCode=0
Oct 30 20:42:02.624830 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:02.624470    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qtr6p" event={"ID":"de36102c-fee1-40dd-ba3f-95691343bae0","Type":"ContainerDied","Data":"8b33ae365c9472b2cd3f5def3d436b433d65027287cba5583046f76423879257"}
Oct 30 20:42:04.661503 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:04.661439    4521 generic.go:356] "Generic (PLEG): container finished" podID="de36102c-fee1-40dd-ba3f-95691343bae0" containerID="22c3d74b327a1231c1466661329eb49a1d534b6cc67d3c540339f008c51b0e36" exitCode=0
Oct 30 20:42:04.662524 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:04.661577    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qtr6p" event={"ID":"de36102c-fee1-40dd-ba3f-95691343bae0","Type":"ContainerDied","Data":"22c3d74b327a1231c1466661329eb49a1d534b6cc67d3c540339f008c51b0e36"}
Oct 30 20:42:04.914612 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:04.914572    4521 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:42:04.914721 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:04.914672    4521 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:42:04.914791 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:04.914721    4521 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:42:04.920890 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:04.920853    4521 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:42:04.921008 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:04.920942    4521 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:42:04.921008 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:04.920989    4521 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:42:05.668964 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:05.668851    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-bckjb" event={"ID":"62149acf-8605-4f8d-b1ac-6e8d6b4865fb","Type":"ContainerStarted","Data":"cba5b42bbb7946b983df7fbc9e84bee3ae8bd1ac22284f6c94104007a8604640"}
Oct 30 20:42:05.671420 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:05.671386    4521 generic.go:356] "Generic (PLEG): container finished" podID="824a2b93-8ec6-4fe1-9d0d-abf8cc679abf" containerID="c9e678524af50be63a33a85e02ab264029c804f1feb8c6215dd5c16ffaecd170" exitCode=0
Oct 30 20:42:05.671559 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:05.671525    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7txbfc" event={"ID":"824a2b93-8ec6-4fe1-9d0d-abf8cc679abf","Type":"ContainerDied","Data":"c9e678524af50be63a33a85e02ab264029c804f1feb8c6215dd5c16ffaecd170"}
Oct 30 20:42:05.673558 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:05.673534    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-bc8769f9-rxf86" event={"ID":"f421589f-1efe-4427-a723-435502b4a14b","Type":"ContainerStarted","Data":"f269c775b8db5b9f44d66764bfdd8de0d57cb6d1c9f5f8b1665a7bb1b7af0121"}
Oct 30 20:42:05.695966 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:05.694731    4521 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-bckjb" podStartSLOduration=2.053577999 podStartE2EDuration="12.694717839s" totalImagesPullingTime="10.64113984s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:41:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:42:05.694061602 +0000 UTC m=+193.039682852" watchObservedRunningTime="2026-10-30 20:42:05.694717839 +0000 UTC m=+193.040339071"
Oct 30 20:42:05.718763 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:05.718706    4521 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-bc8769f9-rxf86" podStartSLOduration=2.183771304 podStartE2EDuration="12.718694312s" totalImagesPullingTime="10.534923008s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:41:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:42:05.717760267 +0000 UTC m=+193.063381508" watchObservedRunningTime="2026-10-30 20:42:05.718694312 +0000 UTC m=+193.064315602"
Oct 30 20:42:05.939964 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:05.939928    4521 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-bckjb"
Oct 30 20:42:06.034170 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:06.034140    4521 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-bc8769f9-rxf86"
Oct 30 20:42:06.038831 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:06.038804    4521 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-bc8769f9-rxf86"
Oct 30 20:42:06.678935 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:06.678901    4521 generic.go:356] "Generic (PLEG): container finished" podID="62d1efb9-0cea-45ca-a468-31f81124bf4b" containerID="68a59453a363ec22cdc40b6c517e9cced11baa2ec2bbc3410ed1b01f46a94928" exitCode=0
Oct 30 20:42:06.679359 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:06.678988    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak7782" event={"ID":"62d1efb9-0cea-45ca-a468-31f81124bf4b","Type":"ContainerDied","Data":"68a59453a363ec22cdc40b6c517e9cced11baa2ec2bbc3410ed1b01f46a94928"}
Oct 30 20:42:06.680482 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:06.680460    4521 generic.go:356] "Generic (PLEG): container finished" podID="54d00e9e-313d-4fde-87a2-e641c4caf568" containerID="11604c963113042e7d881c2e6d14a9a237bc1462d62a92ea7ce52ac29e6f6585" exitCode=0
Oct 30 20:42:06.680562 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:06.680546    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cd4bdk" event={"ID":"54d00e9e-313d-4fde-87a2-e641c4caf568","Type":"ContainerDied","Data":"11604c963113042e7d881c2e6d14a9a237bc1462d62a92ea7ce52ac29e6f6585"}
Oct 30 20:42:06.682208 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:06.682185    4521 generic.go:356] "Generic (PLEG): container finished" podID="1e8b2d49-82b3-47fa-8317-a5df38e00cb4" containerID="5e3fc32a7d1f86d9c572f0a4abd1ce5a06c2a93115f2b5ce94bd307214873497" exitCode=0
Oct 30 20:42:06.682307 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:06.682251    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c984t8" event={"ID":"1e8b2d49-82b3-47fa-8317-a5df38e00cb4","Type":"ContainerDied","Data":"5e3fc32a7d1f86d9c572f0a4abd1ce5a06c2a93115f2b5ce94bd307214873497"}
Oct 30 20:42:06.685720 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:06.685153    4521 generic.go:356] "Generic (PLEG): container finished" podID="fd75cc69-c4d2-4e9f-9215-f0e0b81bd64e" containerID="585d2edd9e51f5ff031e1a88f0c2030da7b305b7a25a5b2bf236ea468f3ef6dd" exitCode=0
Oct 30 20:42:06.685720 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:06.685232    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccd69g" event={"ID":"fd75cc69-c4d2-4e9f-9215-f0e0b81bd64e","Type":"ContainerDied","Data":"585d2edd9e51f5ff031e1a88f0c2030da7b305b7a25a5b2bf236ea468f3ef6dd"}
Oct 30 20:42:06.687519 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:06.687501    4521 generic.go:356] "Generic (PLEG): container finished" podID="747d9d7a-fdca-4839-afe2-ef5933b3cc16" containerID="df51c7df1d6af47cd5808cc50a65401b2bb392799c00a7ea44d7f791bd3bcac1" exitCode=0
Oct 30 20:42:06.687596 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:06.687528    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644l85qk" event={"ID":"747d9d7a-fdca-4839-afe2-ef5933b3cc16","Type":"ContainerDied","Data":"df51c7df1d6af47cd5808cc50a65401b2bb392799c00a7ea44d7f791bd3bcac1"}
Oct 30 20:42:06.694674 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:06.694650    4521 generic.go:356] "Generic (PLEG): container finished" podID="be269a25-d606-4ec1-acb4-38eedf4f201c" containerID="7683d792808cfbadd2af911bebed3681162719d2e4a4560b0a3978a00ed9a5de" exitCode=0
Oct 30 20:42:06.694770 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:06.694750    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644txvp8" event={"ID":"be269a25-d606-4ec1-acb4-38eedf4f201c","Type":"ContainerDied","Data":"7683d792808cfbadd2af911bebed3681162719d2e4a4560b0a3978a00ed9a5de"}
Oct 30 20:42:06.695086 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:06.695068    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-bc8769f9-rxf86"
Oct 30 20:42:06.696030 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:06.696014    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-bc8769f9-rxf86"
Oct 30 20:42:06.946058 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:06.945836    4521 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-bckjb"
Oct 30 20:42:07.549037 ip-10-0-131-125 kubenswrapper[4521]: E1030 20:42:07.549001    4521 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62d1efb9_0cea_45ca_a468_31f81124bf4b.slice/crio-2af19a0fe10f53199726b6354d1ea0ecc78010e08c90996b7c3a5a1879e3cc4a.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 20:42:07.704453 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:07.704418    4521 generic.go:356] "Generic (PLEG): container finished" podID="824a2b93-8ec6-4fe1-9d0d-abf8cc679abf" containerID="9053727f49b2e3260f10d565b647c7d180c6180a82d3306d9c8530fe743c5637" exitCode=0
Oct 30 20:42:07.704893 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:07.704496    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7txbfc" event={"ID":"824a2b93-8ec6-4fe1-9d0d-abf8cc679abf","Type":"ContainerDied","Data":"9053727f49b2e3260f10d565b647c7d180c6180a82d3306d9c8530fe743c5637"}
Oct 30 20:42:07.712047 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:07.712019    4521 generic.go:356] "Generic (PLEG): container finished" podID="62d1efb9-0cea-45ca-a468-31f81124bf4b" containerID="2af19a0fe10f53199726b6354d1ea0ecc78010e08c90996b7c3a5a1879e3cc4a" exitCode=0
Oct 30 20:42:07.712165 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:07.712127    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak7782" event={"ID":"62d1efb9-0cea-45ca-a468-31f81124bf4b","Type":"ContainerDied","Data":"2af19a0fe10f53199726b6354d1ea0ecc78010e08c90996b7c3a5a1879e3cc4a"}
Oct 30 20:42:07.718729 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:07.718167    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cd4bdk" event={"ID":"54d00e9e-313d-4fde-87a2-e641c4caf568","Type":"ContainerStarted","Data":"14c21e638a1800222645370887ac7fdc0852c1b06869269dded9d7fdb2b23b9a"}
Oct 30 20:42:07.722153 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:07.722124    4521 generic.go:356] "Generic (PLEG): container finished" podID="3963bd73-a6f8-4650-a9ef-8ffd1a2ee63f" containerID="4cc3b7b0b53ae454a62cd1a7271a71207003243744cccf18e6644f76111cd2f6" exitCode=0
Oct 30 20:42:07.722264 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:07.722204    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cpdbc" event={"ID":"3963bd73-a6f8-4650-a9ef-8ffd1a2ee63f","Type":"ContainerDied","Data":"4cc3b7b0b53ae454a62cd1a7271a71207003243744cccf18e6644f76111cd2f6"}
Oct 30 20:42:07.726038 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:07.725438    4521 generic.go:356] "Generic (PLEG): container finished" podID="fd75cc69-c4d2-4e9f-9215-f0e0b81bd64e" containerID="645243b54157455b1c3360cb2baf083ece43fb0242a0fd2019f60d0f808da149" exitCode=0
Oct 30 20:42:07.726038 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:07.725521    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccd69g" event={"ID":"fd75cc69-c4d2-4e9f-9215-f0e0b81bd64e","Type":"ContainerDied","Data":"645243b54157455b1c3360cb2baf083ece43fb0242a0fd2019f60d0f808da149"}
Oct 30 20:42:07.731921 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:07.731894    4521 generic.go:356] "Generic (PLEG): container finished" podID="747d9d7a-fdca-4839-afe2-ef5933b3cc16" containerID="9c3fc5a7e71cd8b1c3be14584628a410f9a6e963193eaf8d67d42a715a13fbb9" exitCode=0
Oct 30 20:42:07.732029 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:07.731957    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644l85qk" event={"ID":"747d9d7a-fdca-4839-afe2-ef5933b3cc16","Type":"ContainerDied","Data":"9c3fc5a7e71cd8b1c3be14584628a410f9a6e963193eaf8d67d42a715a13fbb9"}
Oct 30 20:42:07.732396 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:07.732376    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-bckjb"
Oct 30 20:42:07.733491 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:07.733401    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-bckjb"
Oct 30 20:42:08.776755 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:08.776712    4521 generic.go:356] "Generic (PLEG): container finished" podID="54d00e9e-313d-4fde-87a2-e641c4caf568" containerID="14c21e638a1800222645370887ac7fdc0852c1b06869269dded9d7fdb2b23b9a" exitCode=0
Oct 30 20:42:08.777569 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:08.777543    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cd4bdk" event={"ID":"54d00e9e-313d-4fde-87a2-e641c4caf568","Type":"ContainerDied","Data":"14c21e638a1800222645370887ac7fdc0852c1b06869269dded9d7fdb2b23b9a"}
Oct 30 20:42:09.796193 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:09.796147    4521 generic.go:356] "Generic (PLEG): container finished" podID="3963bd73-a6f8-4650-a9ef-8ffd1a2ee63f" containerID="58a6b001c27845aee10bdd67eb3759b525ea6384b7d8d6cb4054f17ae0ac1dae" exitCode=0
Oct 30 20:42:09.797505 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:09.797449    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cpdbc" event={"ID":"3963bd73-a6f8-4650-a9ef-8ffd1a2ee63f","Type":"ContainerDied","Data":"58a6b001c27845aee10bdd67eb3759b525ea6384b7d8d6cb4054f17ae0ac1dae"}
Oct 30 20:42:16.483356 ip-10-0-131-125 kubenswrapper[4521]: E1030 20:42:16.483235    4521 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lkn2k" containerName="extractor"
Oct 30 20:42:24.961057 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:24.961010    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c984t8" event={"ID":"1e8b2d49-82b3-47fa-8317-a5df38e00cb4","Type":"ContainerStarted","Data":"d6962425c309b1a35f693f541103e2e76eca0d57037b487651982d1c6d330a58"}
Oct 30 20:42:24.964831 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:24.964682    4521 generic.go:356] "Generic (PLEG): container finished" podID="de36102c-fee1-40dd-ba3f-95691343bae0" containerID="13287d17b18ff8d9fdd927c6cf53f6e806f8b1a25e41edf4d9617e6bf2822d83" exitCode=0
Oct 30 20:42:24.964831 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:24.964777    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qtr6p" event={"ID":"de36102c-fee1-40dd-ba3f-95691343bae0","Type":"ContainerDied","Data":"13287d17b18ff8d9fdd927c6cf53f6e806f8b1a25e41edf4d9617e6bf2822d83"}
Oct 30 20:42:24.972312 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:24.972063    4521 generic.go:356] "Generic (PLEG): container finished" podID="be269a25-d606-4ec1-acb4-38eedf4f201c" containerID="a4e60bac39e016458a048ae5191077f64781113041ef5869cad9ad7367611817" exitCode=0
Oct 30 20:42:24.972312 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:24.972143    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644txvp8" event={"ID":"be269a25-d606-4ec1-acb4-38eedf4f201c","Type":"ContainerDied","Data":"a4e60bac39e016458a048ae5191077f64781113041ef5869cad9ad7367611817"}
Oct 30 20:42:24.986726 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:24.986662    4521 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c984t8" podStartSLOduration=2.947850572 podStartE2EDuration="37.986642627s" totalImagesPullingTime="35.038792055s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:41:47 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:42:24.9834574 +0000 UTC m=+212.329078666" watchObservedRunningTime="2026-10-30 20:42:24.986642627 +0000 UTC m=+212.332263867"
Oct 30 20:42:25.984233 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:25.981550    4521 generic.go:356] "Generic (PLEG): container finished" podID="1e8b2d49-82b3-47fa-8317-a5df38e00cb4" containerID="d6962425c309b1a35f693f541103e2e76eca0d57037b487651982d1c6d330a58" exitCode=0
Oct 30 20:42:25.984233 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:25.982394    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c984t8" event={"ID":"1e8b2d49-82b3-47fa-8317-a5df38e00cb4","Type":"ContainerDied","Data":"d6962425c309b1a35f693f541103e2e76eca0d57037b487651982d1c6d330a58"}
Oct 30 20:42:26.186396 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:26.186340    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qtr6p"
Oct 30 20:42:26.200784 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:26.200743    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644txvp8"
Oct 30 20:42:26.284800 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:26.284765    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be269a25-d606-4ec1-acb4-38eedf4f201c-kube-api-access-6nph6\" (UniqueName: \"kubernetes.io/projected/be269a25-d606-4ec1-acb4-38eedf4f201c-kube-api-access-6nph6\") pod \"be269a25-d606-4ec1-acb4-38eedf4f201c\" (UID: \"be269a25-d606-4ec1-acb4-38eedf4f201c\") "
Oct 30 20:42:26.284965 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:26.284827    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be269a25-d606-4ec1-acb4-38eedf4f201c-util\" (UniqueName: \"kubernetes.io/empty-dir/be269a25-d606-4ec1-acb4-38eedf4f201c-util\") pod \"be269a25-d606-4ec1-acb4-38eedf4f201c\" (UID: \"be269a25-d606-4ec1-acb4-38eedf4f201c\") "
Oct 30 20:42:26.284965 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:26.284867    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de36102c-fee1-40dd-ba3f-95691343bae0-kube-api-access-pvnlk\" (UniqueName: \"kubernetes.io/projected/de36102c-fee1-40dd-ba3f-95691343bae0-kube-api-access-pvnlk\") pod \"de36102c-fee1-40dd-ba3f-95691343bae0\" (UID: \"de36102c-fee1-40dd-ba3f-95691343bae0\") "
Oct 30 20:42:26.284965 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:26.284895    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de36102c-fee1-40dd-ba3f-95691343bae0-util\" (UniqueName: \"kubernetes.io/empty-dir/de36102c-fee1-40dd-ba3f-95691343bae0-util\") pod \"de36102c-fee1-40dd-ba3f-95691343bae0\" (UID: \"de36102c-fee1-40dd-ba3f-95691343bae0\") "
Oct 30 20:42:26.284965 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:26.284939    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be269a25-d606-4ec1-acb4-38eedf4f201c-bundle\" (UniqueName: \"kubernetes.io/empty-dir/be269a25-d606-4ec1-acb4-38eedf4f201c-bundle\") pod \"be269a25-d606-4ec1-acb4-38eedf4f201c\" (UID: \"be269a25-d606-4ec1-acb4-38eedf4f201c\") "
Oct 30 20:42:26.285188 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:26.285006    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de36102c-fee1-40dd-ba3f-95691343bae0-bundle\" (UniqueName: \"kubernetes.io/empty-dir/de36102c-fee1-40dd-ba3f-95691343bae0-bundle\") pod \"de36102c-fee1-40dd-ba3f-95691343bae0\" (UID: \"de36102c-fee1-40dd-ba3f-95691343bae0\") "
Oct 30 20:42:26.286034 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:26.286001    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de36102c-fee1-40dd-ba3f-95691343bae0-bundle" pod "de36102c-fee1-40dd-ba3f-95691343bae0" (UID: "de36102c-fee1-40dd-ba3f-95691343bae0"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:42:26.287373 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:26.287337    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be269a25-d606-4ec1-acb4-38eedf4f201c-bundle" pod "be269a25-d606-4ec1-acb4-38eedf4f201c" (UID: "be269a25-d606-4ec1-acb4-38eedf4f201c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:42:26.287955 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:26.287928    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be269a25-d606-4ec1-acb4-38eedf4f201c-kube-api-access-6nph6" pod "be269a25-d606-4ec1-acb4-38eedf4f201c" (UID: "be269a25-d606-4ec1-acb4-38eedf4f201c"). InnerVolumeSpecName "kube-api-access-6nph6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:42:26.288529 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:26.288502    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de36102c-fee1-40dd-ba3f-95691343bae0-kube-api-access-pvnlk" pod "de36102c-fee1-40dd-ba3f-95691343bae0" (UID: "de36102c-fee1-40dd-ba3f-95691343bae0"). InnerVolumeSpecName "kube-api-access-pvnlk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:42:26.290108 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:26.290079    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de36102c-fee1-40dd-ba3f-95691343bae0-util" pod "de36102c-fee1-40dd-ba3f-95691343bae0" (UID: "de36102c-fee1-40dd-ba3f-95691343bae0"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:42:26.291916 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:26.291883    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be269a25-d606-4ec1-acb4-38eedf4f201c-util" pod "be269a25-d606-4ec1-acb4-38eedf4f201c" (UID: "be269a25-d606-4ec1-acb4-38eedf4f201c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:42:26.385958 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:26.385920    4521 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/de36102c-fee1-40dd-ba3f-95691343bae0-bundle\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:42:26.385958 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:26.385954    4521 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6nph6\" (UniqueName: \"kubernetes.io/projected/be269a25-d606-4ec1-acb4-38eedf4f201c-kube-api-access-6nph6\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:42:26.385958 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:26.385965    4521 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/be269a25-d606-4ec1-acb4-38eedf4f201c-util\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:42:26.386192 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:26.385974    4521 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvnlk\" (UniqueName: \"kubernetes.io/projected/de36102c-fee1-40dd-ba3f-95691343bae0-kube-api-access-pvnlk\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:42:26.386192 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:26.385984    4521 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/de36102c-fee1-40dd-ba3f-95691343bae0-util\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:42:26.386192 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:26.385992    4521 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/be269a25-d606-4ec1-acb4-38eedf4f201c-bundle\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:42:26.990101 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:26.990066    4521 generic.go:356] "Generic (PLEG): container finished" podID="fd75cc69-c4d2-4e9f-9215-f0e0b81bd64e" containerID="0616a60cedceedc68d87388563b91119a19f48a17249accf15ea4a1398a7ae64" exitCode=0
Oct 30 20:42:26.990600 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:26.990157    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccd69g" event={"ID":"fd75cc69-c4d2-4e9f-9215-f0e0b81bd64e","Type":"ContainerDied","Data":"0616a60cedceedc68d87388563b91119a19f48a17249accf15ea4a1398a7ae64"}
Oct 30 20:42:26.992644 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:26.992618    4521 generic.go:356] "Generic (PLEG): container finished" podID="747d9d7a-fdca-4839-afe2-ef5933b3cc16" containerID="4695b0e8fc0697cde58af1f398cff4bfc6db9abbaafcb7dfad187b582253d7a1" exitCode=0
Oct 30 20:42:26.992764 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:26.992704    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644l85qk" event={"ID":"747d9d7a-fdca-4839-afe2-ef5933b3cc16","Type":"ContainerDied","Data":"4695b0e8fc0697cde58af1f398cff4bfc6db9abbaafcb7dfad187b582253d7a1"}
Oct 30 20:42:26.994997 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:26.994977    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qtr6p"
Oct 30 20:42:26.995103 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:26.994979    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qtr6p" event={"ID":"de36102c-fee1-40dd-ba3f-95691343bae0","Type":"ContainerDied","Data":"7aa97f4876e218bce99399f7e39f17fd5bd6fdf4fd5e0cc99b27dde58286a3a6"}
Oct 30 20:42:26.995170 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:26.995113    4521 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7aa97f4876e218bce99399f7e39f17fd5bd6fdf4fd5e0cc99b27dde58286a3a6"
Oct 30 20:42:26.997631 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:26.997115    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644txvp8" event={"ID":"be269a25-d606-4ec1-acb4-38eedf4f201c","Type":"ContainerDied","Data":"aa31a4830233f3567bd5e73bc6d245157b06f95e2a3ce30a3e6ed99cdd11088d"}
Oct 30 20:42:26.997631 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:26.997154    4521 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aa31a4830233f3567bd5e73bc6d245157b06f95e2a3ce30a3e6ed99cdd11088d"
Oct 30 20:42:26.997631 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:26.997124    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644txvp8"
Oct 30 20:42:26.999751 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:26.999733    4521 generic.go:356] "Generic (PLEG): container finished" podID="824a2b93-8ec6-4fe1-9d0d-abf8cc679abf" containerID="d335fc314f7ec2d1e448c97b2ae489379b3d6027c2b3f3e164cae1faaa6921e5" exitCode=0
Oct 30 20:42:26.999839 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:26.999798    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7txbfc" event={"ID":"824a2b93-8ec6-4fe1-9d0d-abf8cc679abf","Type":"ContainerDied","Data":"d335fc314f7ec2d1e448c97b2ae489379b3d6027c2b3f3e164cae1faaa6921e5"}
Oct 30 20:42:27.001934 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:27.001912    4521 generic.go:356] "Generic (PLEG): container finished" podID="62d1efb9-0cea-45ca-a468-31f81124bf4b" containerID="6672cf1d5d35519cbfeef3d8136e0ce8dc425d760b46ff4b73cfb9ed4d0e1d42" exitCode=0
Oct 30 20:42:27.002023 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:27.001999    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak7782" event={"ID":"62d1efb9-0cea-45ca-a468-31f81124bf4b","Type":"ContainerDied","Data":"6672cf1d5d35519cbfeef3d8136e0ce8dc425d760b46ff4b73cfb9ed4d0e1d42"}
Oct 30 20:42:27.132163 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:27.132137    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c984t8"
Oct 30 20:42:27.193184 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:27.193147    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e8b2d49-82b3-47fa-8317-a5df38e00cb4-util\" (UniqueName: \"kubernetes.io/empty-dir/1e8b2d49-82b3-47fa-8317-a5df38e00cb4-util\") pod \"1e8b2d49-82b3-47fa-8317-a5df38e00cb4\" (UID: \"1e8b2d49-82b3-47fa-8317-a5df38e00cb4\") "
Oct 30 20:42:27.193369 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:27.193205    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e8b2d49-82b3-47fa-8317-a5df38e00cb4-kube-api-access-zngzx\" (UniqueName: \"kubernetes.io/projected/1e8b2d49-82b3-47fa-8317-a5df38e00cb4-kube-api-access-zngzx\") pod \"1e8b2d49-82b3-47fa-8317-a5df38e00cb4\" (UID: \"1e8b2d49-82b3-47fa-8317-a5df38e00cb4\") "
Oct 30 20:42:27.193467 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:27.193448    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e8b2d49-82b3-47fa-8317-a5df38e00cb4-bundle\" (UniqueName: \"kubernetes.io/empty-dir/1e8b2d49-82b3-47fa-8317-a5df38e00cb4-bundle\") pod \"1e8b2d49-82b3-47fa-8317-a5df38e00cb4\" (UID: \"1e8b2d49-82b3-47fa-8317-a5df38e00cb4\") "
Oct 30 20:42:27.194027 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:27.194002    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e8b2d49-82b3-47fa-8317-a5df38e00cb4-bundle" pod "1e8b2d49-82b3-47fa-8317-a5df38e00cb4" (UID: "1e8b2d49-82b3-47fa-8317-a5df38e00cb4"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:42:27.195275 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:27.195257    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e8b2d49-82b3-47fa-8317-a5df38e00cb4-kube-api-access-zngzx" pod "1e8b2d49-82b3-47fa-8317-a5df38e00cb4" (UID: "1e8b2d49-82b3-47fa-8317-a5df38e00cb4"). InnerVolumeSpecName "kube-api-access-zngzx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:42:27.199410 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:27.199384    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e8b2d49-82b3-47fa-8317-a5df38e00cb4-util" pod "1e8b2d49-82b3-47fa-8317-a5df38e00cb4" (UID: "1e8b2d49-82b3-47fa-8317-a5df38e00cb4"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:42:27.294455 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:27.294426    4521 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1e8b2d49-82b3-47fa-8317-a5df38e00cb4-bundle\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:42:27.294455 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:27.294452    4521 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1e8b2d49-82b3-47fa-8317-a5df38e00cb4-util\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:42:27.294455 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:27.294461    4521 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zngzx\" (UniqueName: \"kubernetes.io/projected/1e8b2d49-82b3-47fa-8317-a5df38e00cb4-kube-api-access-zngzx\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:42:28.007902 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:28.007816    4521 generic.go:356] "Generic (PLEG): container finished" podID="54d00e9e-313d-4fde-87a2-e641c4caf568" containerID="2b7b529777e364325331aa3ae1f5827264153d38da04af07d78e6c10c48f898e" exitCode=0
Oct 30 20:42:28.008337 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:28.007903    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cd4bdk" event={"ID":"54d00e9e-313d-4fde-87a2-e641c4caf568","Type":"ContainerDied","Data":"2b7b529777e364325331aa3ae1f5827264153d38da04af07d78e6c10c48f898e"}
Oct 30 20:42:28.009916 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:28.009894    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c984t8"
Oct 30 20:42:28.010045 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:28.009892    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c984t8" event={"ID":"1e8b2d49-82b3-47fa-8317-a5df38e00cb4","Type":"ContainerDied","Data":"c389e40b7d234b3d0ee586824e1da205d2e1503cfac70c3ca685b43ea42de7b3"}
Oct 30 20:42:28.010045 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:28.009987    4521 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c389e40b7d234b3d0ee586824e1da205d2e1503cfac70c3ca685b43ea42de7b3"
Oct 30 20:42:28.012147 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:28.012121    4521 generic.go:356] "Generic (PLEG): container finished" podID="3963bd73-a6f8-4650-a9ef-8ffd1a2ee63f" containerID="4c765ab4e408cffbfd9c068aae8a1b58f52969848630e5680d09f8fb4fc0cc06" exitCode=0
Oct 30 20:42:28.012259 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:28.012169    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cpdbc" event={"ID":"3963bd73-a6f8-4650-a9ef-8ffd1a2ee63f","Type":"ContainerDied","Data":"4c765ab4e408cffbfd9c068aae8a1b58f52969848630e5680d09f8fb4fc0cc06"}
Oct 30 20:42:28.151309 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:28.151282    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccd69g"
Oct 30 20:42:28.193842 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:28.193812    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7txbfc"
Oct 30 20:42:28.203028 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:28.202992    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd75cc69-c4d2-4e9f-9215-f0e0b81bd64e-util\" (UniqueName: \"kubernetes.io/empty-dir/fd75cc69-c4d2-4e9f-9215-f0e0b81bd64e-util\") pod \"fd75cc69-c4d2-4e9f-9215-f0e0b81bd64e\" (UID: \"fd75cc69-c4d2-4e9f-9215-f0e0b81bd64e\") "
Oct 30 20:42:28.203177 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:28.203047    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd75cc69-c4d2-4e9f-9215-f0e0b81bd64e-bundle\" (UniqueName: \"kubernetes.io/empty-dir/fd75cc69-c4d2-4e9f-9215-f0e0b81bd64e-bundle\") pod \"fd75cc69-c4d2-4e9f-9215-f0e0b81bd64e\" (UID: \"fd75cc69-c4d2-4e9f-9215-f0e0b81bd64e\") "
Oct 30 20:42:28.203177 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:28.203081    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd75cc69-c4d2-4e9f-9215-f0e0b81bd64e-kube-api-access-7m2mf\" (UniqueName: \"kubernetes.io/projected/fd75cc69-c4d2-4e9f-9215-f0e0b81bd64e-kube-api-access-7m2mf\") pod \"fd75cc69-c4d2-4e9f-9215-f0e0b81bd64e\" (UID: \"fd75cc69-c4d2-4e9f-9215-f0e0b81bd64e\") "
Oct 30 20:42:28.203862 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:28.203834    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd75cc69-c4d2-4e9f-9215-f0e0b81bd64e-bundle" pod "fd75cc69-c4d2-4e9f-9215-f0e0b81bd64e" (UID: "fd75cc69-c4d2-4e9f-9215-f0e0b81bd64e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:42:28.205362 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:28.205302    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd75cc69-c4d2-4e9f-9215-f0e0b81bd64e-kube-api-access-7m2mf" pod "fd75cc69-c4d2-4e9f-9215-f0e0b81bd64e" (UID: "fd75cc69-c4d2-4e9f-9215-f0e0b81bd64e"). InnerVolumeSpecName "kube-api-access-7m2mf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:42:28.209003 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:28.208976    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd75cc69-c4d2-4e9f-9215-f0e0b81bd64e-util" pod "fd75cc69-c4d2-4e9f-9215-f0e0b81bd64e" (UID: "fd75cc69-c4d2-4e9f-9215-f0e0b81bd64e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:42:28.213175 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:28.213155    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak7782"
Oct 30 20:42:28.215403 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:28.215381    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644l85qk"
Oct 30 20:42:28.303831 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:28.303747    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/747d9d7a-fdca-4839-afe2-ef5933b3cc16-kube-api-access-h5nvh\" (UniqueName: \"kubernetes.io/projected/747d9d7a-fdca-4839-afe2-ef5933b3cc16-kube-api-access-h5nvh\") pod \"747d9d7a-fdca-4839-afe2-ef5933b3cc16\" (UID: \"747d9d7a-fdca-4839-afe2-ef5933b3cc16\") "
Oct 30 20:42:28.303831 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:28.303788    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/824a2b93-8ec6-4fe1-9d0d-abf8cc679abf-bundle\" (UniqueName: \"kubernetes.io/empty-dir/824a2b93-8ec6-4fe1-9d0d-abf8cc679abf-bundle\") pod \"824a2b93-8ec6-4fe1-9d0d-abf8cc679abf\" (UID: \"824a2b93-8ec6-4fe1-9d0d-abf8cc679abf\") "
Oct 30 20:42:28.303831 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:28.303814    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62d1efb9-0cea-45ca-a468-31f81124bf4b-bundle\" (UniqueName: \"kubernetes.io/empty-dir/62d1efb9-0cea-45ca-a468-31f81124bf4b-bundle\") pod \"62d1efb9-0cea-45ca-a468-31f81124bf4b\" (UID: \"62d1efb9-0cea-45ca-a468-31f81124bf4b\") "
Oct 30 20:42:28.304123 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:28.303834    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/824a2b93-8ec6-4fe1-9d0d-abf8cc679abf-kube-api-access-zfm7n\" (UniqueName: \"kubernetes.io/projected/824a2b93-8ec6-4fe1-9d0d-abf8cc679abf-kube-api-access-zfm7n\") pod \"824a2b93-8ec6-4fe1-9d0d-abf8cc679abf\" (UID: \"824a2b93-8ec6-4fe1-9d0d-abf8cc679abf\") "
Oct 30 20:42:28.304123 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:28.303904    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/747d9d7a-fdca-4839-afe2-ef5933b3cc16-bundle\" (UniqueName: \"kubernetes.io/empty-dir/747d9d7a-fdca-4839-afe2-ef5933b3cc16-bundle\") pod \"747d9d7a-fdca-4839-afe2-ef5933b3cc16\" (UID: \"747d9d7a-fdca-4839-afe2-ef5933b3cc16\") "
Oct 30 20:42:28.304123 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:28.303940    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/824a2b93-8ec6-4fe1-9d0d-abf8cc679abf-util\" (UniqueName: \"kubernetes.io/empty-dir/824a2b93-8ec6-4fe1-9d0d-abf8cc679abf-util\") pod \"824a2b93-8ec6-4fe1-9d0d-abf8cc679abf\" (UID: \"824a2b93-8ec6-4fe1-9d0d-abf8cc679abf\") "
Oct 30 20:42:28.304123 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:28.303973    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/747d9d7a-fdca-4839-afe2-ef5933b3cc16-util\" (UniqueName: \"kubernetes.io/empty-dir/747d9d7a-fdca-4839-afe2-ef5933b3cc16-util\") pod \"747d9d7a-fdca-4839-afe2-ef5933b3cc16\" (UID: \"747d9d7a-fdca-4839-afe2-ef5933b3cc16\") "
Oct 30 20:42:28.304123 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:28.304008    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62d1efb9-0cea-45ca-a468-31f81124bf4b-util\" (UniqueName: \"kubernetes.io/empty-dir/62d1efb9-0cea-45ca-a468-31f81124bf4b-util\") pod \"62d1efb9-0cea-45ca-a468-31f81124bf4b\" (UID: \"62d1efb9-0cea-45ca-a468-31f81124bf4b\") "
Oct 30 20:42:28.304123 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:28.304045    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62d1efb9-0cea-45ca-a468-31f81124bf4b-kube-api-access-mq647\" (UniqueName: \"kubernetes.io/projected/62d1efb9-0cea-45ca-a468-31f81124bf4b-kube-api-access-mq647\") pod \"62d1efb9-0cea-45ca-a468-31f81124bf4b\" (UID: \"62d1efb9-0cea-45ca-a468-31f81124bf4b\") "
Oct 30 20:42:28.304434 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:28.304277    4521 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7m2mf\" (UniqueName: \"kubernetes.io/projected/fd75cc69-c4d2-4e9f-9215-f0e0b81bd64e-kube-api-access-7m2mf\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:42:28.304434 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:28.304297    4521 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fd75cc69-c4d2-4e9f-9215-f0e0b81bd64e-util\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:42:28.304434 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:28.304379    4521 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fd75cc69-c4d2-4e9f-9215-f0e0b81bd64e-bundle\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:42:28.304573 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:28.304480    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/824a2b93-8ec6-4fe1-9d0d-abf8cc679abf-bundle" pod "824a2b93-8ec6-4fe1-9d0d-abf8cc679abf" (UID: "824a2b93-8ec6-4fe1-9d0d-abf8cc679abf"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:42:28.304573 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:28.304481    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/747d9d7a-fdca-4839-afe2-ef5933b3cc16-bundle" pod "747d9d7a-fdca-4839-afe2-ef5933b3cc16" (UID: "747d9d7a-fdca-4839-afe2-ef5933b3cc16"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:42:28.304815 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:28.304790    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62d1efb9-0cea-45ca-a468-31f81124bf4b-bundle" pod "62d1efb9-0cea-45ca-a468-31f81124bf4b" (UID: "62d1efb9-0cea-45ca-a468-31f81124bf4b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:42:28.306945 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:28.306915    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/747d9d7a-fdca-4839-afe2-ef5933b3cc16-kube-api-access-h5nvh" pod "747d9d7a-fdca-4839-afe2-ef5933b3cc16" (UID: "747d9d7a-fdca-4839-afe2-ef5933b3cc16"). InnerVolumeSpecName "kube-api-access-h5nvh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:42:28.307221 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:28.307189    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62d1efb9-0cea-45ca-a468-31f81124bf4b-kube-api-access-mq647" pod "62d1efb9-0cea-45ca-a468-31f81124bf4b" (UID: "62d1efb9-0cea-45ca-a468-31f81124bf4b"). InnerVolumeSpecName "kube-api-access-mq647". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:42:28.307449 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:28.307431    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/824a2b93-8ec6-4fe1-9d0d-abf8cc679abf-kube-api-access-zfm7n" pod "824a2b93-8ec6-4fe1-9d0d-abf8cc679abf" (UID: "824a2b93-8ec6-4fe1-9d0d-abf8cc679abf"). InnerVolumeSpecName "kube-api-access-zfm7n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:42:28.310856 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:28.310834    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/747d9d7a-fdca-4839-afe2-ef5933b3cc16-util" pod "747d9d7a-fdca-4839-afe2-ef5933b3cc16" (UID: "747d9d7a-fdca-4839-afe2-ef5933b3cc16"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:42:28.314404 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:28.314382    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/824a2b93-8ec6-4fe1-9d0d-abf8cc679abf-util" pod "824a2b93-8ec6-4fe1-9d0d-abf8cc679abf" (UID: "824a2b93-8ec6-4fe1-9d0d-abf8cc679abf"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:42:28.314464 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:28.314445    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62d1efb9-0cea-45ca-a468-31f81124bf4b-util" pod "62d1efb9-0cea-45ca-a468-31f81124bf4b" (UID: "62d1efb9-0cea-45ca-a468-31f81124bf4b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:42:28.405030 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:28.404990    4521 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5nvh\" (UniqueName: \"kubernetes.io/projected/747d9d7a-fdca-4839-afe2-ef5933b3cc16-kube-api-access-h5nvh\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:42:28.405030 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:28.405023    4521 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/824a2b93-8ec6-4fe1-9d0d-abf8cc679abf-bundle\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:42:28.405030 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:28.405033    4521 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/62d1efb9-0cea-45ca-a468-31f81124bf4b-bundle\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:42:28.405240 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:28.405042    4521 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfm7n\" (UniqueName: \"kubernetes.io/projected/824a2b93-8ec6-4fe1-9d0d-abf8cc679abf-kube-api-access-zfm7n\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:42:28.405240 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:28.405051    4521 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/747d9d7a-fdca-4839-afe2-ef5933b3cc16-bundle\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:42:28.405240 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:28.405059    4521 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/824a2b93-8ec6-4fe1-9d0d-abf8cc679abf-util\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:42:28.405240 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:28.405066    4521 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/747d9d7a-fdca-4839-afe2-ef5933b3cc16-util\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:42:28.405240 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:28.405073    4521 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/62d1efb9-0cea-45ca-a468-31f81124bf4b-util\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:42:28.405240 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:28.405080    4521 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mq647\" (UniqueName: \"kubernetes.io/projected/62d1efb9-0cea-45ca-a468-31f81124bf4b-kube-api-access-mq647\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:42:29.018609 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:29.018574    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccd69g" event={"ID":"fd75cc69-c4d2-4e9f-9215-f0e0b81bd64e","Type":"ContainerDied","Data":"e5fb688f9cf8a50423bc911570d09a09f00f3b515c5521c303fe33c1d06eca3c"}
Oct 30 20:42:29.019037 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:29.018625    4521 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e5fb688f9cf8a50423bc911570d09a09f00f3b515c5521c303fe33c1d06eca3c"
Oct 30 20:42:29.019037 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:29.018581    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccd69g"
Oct 30 20:42:29.020720 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:29.020698    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644l85qk"
Oct 30 20:42:29.020847 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:29.020744    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644l85qk" event={"ID":"747d9d7a-fdca-4839-afe2-ef5933b3cc16","Type":"ContainerDied","Data":"d98d59991570206c9ef411cfedb636efc1e8b458698e5a5784c767245ce06280"}
Oct 30 20:42:29.020847 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:29.020784    4521 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d98d59991570206c9ef411cfedb636efc1e8b458698e5a5784c767245ce06280"
Oct 30 20:42:29.023093 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:29.023072    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7txbfc" event={"ID":"824a2b93-8ec6-4fe1-9d0d-abf8cc679abf","Type":"ContainerDied","Data":"2c2bbc95293c77b15523a1f3d6668e96c8b8f97a6ffbc20c09b7f5f49f952958"}
Oct 30 20:42:29.023189 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:29.023104    4521 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2c2bbc95293c77b15523a1f3d6668e96c8b8f97a6ffbc20c09b7f5f49f952958"
Oct 30 20:42:29.023189 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:29.023135    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7txbfc"
Oct 30 20:42:29.025253 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:29.025156    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak7782"
Oct 30 20:42:29.025253 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:29.025193    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak7782" event={"ID":"62d1efb9-0cea-45ca-a468-31f81124bf4b","Type":"ContainerDied","Data":"865a529ec5ebc9de8190e5c933563a4d8b6a1b77d3bf9fe2f7eb66621aadee07"}
Oct 30 20:42:29.025253 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:29.025233    4521 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="865a529ec5ebc9de8190e5c933563a4d8b6a1b77d3bf9fe2f7eb66621aadee07"
Oct 30 20:42:29.162155 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:29.162125    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cd4bdk"
Oct 30 20:42:29.169704 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:29.169686    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cpdbc"
Oct 30 20:42:29.210534 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:29.210508    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3963bd73-a6f8-4650-a9ef-8ffd1a2ee63f-util\" (UniqueName: \"kubernetes.io/empty-dir/3963bd73-a6f8-4650-a9ef-8ffd1a2ee63f-util\") pod \"3963bd73-a6f8-4650-a9ef-8ffd1a2ee63f\" (UID: \"3963bd73-a6f8-4650-a9ef-8ffd1a2ee63f\") "
Oct 30 20:42:29.210664 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:29.210569    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54d00e9e-313d-4fde-87a2-e641c4caf568-kube-api-access-dnhh8\" (UniqueName: \"kubernetes.io/projected/54d00e9e-313d-4fde-87a2-e641c4caf568-kube-api-access-dnhh8\") pod \"54d00e9e-313d-4fde-87a2-e641c4caf568\" (UID: \"54d00e9e-313d-4fde-87a2-e641c4caf568\") "
Oct 30 20:42:29.210664 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:29.210634    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54d00e9e-313d-4fde-87a2-e641c4caf568-util\" (UniqueName: \"kubernetes.io/empty-dir/54d00e9e-313d-4fde-87a2-e641c4caf568-util\") pod \"54d00e9e-313d-4fde-87a2-e641c4caf568\" (UID: \"54d00e9e-313d-4fde-87a2-e641c4caf568\") "
Oct 30 20:42:29.210664 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:29.210656    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54d00e9e-313d-4fde-87a2-e641c4caf568-bundle\" (UniqueName: \"kubernetes.io/empty-dir/54d00e9e-313d-4fde-87a2-e641c4caf568-bundle\") pod \"54d00e9e-313d-4fde-87a2-e641c4caf568\" (UID: \"54d00e9e-313d-4fde-87a2-e641c4caf568\") "
Oct 30 20:42:29.210856 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:29.210689    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3963bd73-a6f8-4650-a9ef-8ffd1a2ee63f-kube-api-access-l9dtj\" (UniqueName: \"kubernetes.io/projected/3963bd73-a6f8-4650-a9ef-8ffd1a2ee63f-kube-api-access-l9dtj\") pod \"3963bd73-a6f8-4650-a9ef-8ffd1a2ee63f\" (UID: \"3963bd73-a6f8-4650-a9ef-8ffd1a2ee63f\") "
Oct 30 20:42:29.210856 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:29.210732    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3963bd73-a6f8-4650-a9ef-8ffd1a2ee63f-bundle\" (UniqueName: \"kubernetes.io/empty-dir/3963bd73-a6f8-4650-a9ef-8ffd1a2ee63f-bundle\") pod \"3963bd73-a6f8-4650-a9ef-8ffd1a2ee63f\" (UID: \"3963bd73-a6f8-4650-a9ef-8ffd1a2ee63f\") "
Oct 30 20:42:29.211358 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:29.211306    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3963bd73-a6f8-4650-a9ef-8ffd1a2ee63f-bundle" pod "3963bd73-a6f8-4650-a9ef-8ffd1a2ee63f" (UID: "3963bd73-a6f8-4650-a9ef-8ffd1a2ee63f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:42:29.211466 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:29.211356    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54d00e9e-313d-4fde-87a2-e641c4caf568-bundle" pod "54d00e9e-313d-4fde-87a2-e641c4caf568" (UID: "54d00e9e-313d-4fde-87a2-e641c4caf568"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:42:29.212912 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:29.212882    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54d00e9e-313d-4fde-87a2-e641c4caf568-kube-api-access-dnhh8" pod "54d00e9e-313d-4fde-87a2-e641c4caf568" (UID: "54d00e9e-313d-4fde-87a2-e641c4caf568"). InnerVolumeSpecName "kube-api-access-dnhh8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:42:29.214160 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:29.214134    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3963bd73-a6f8-4650-a9ef-8ffd1a2ee63f-kube-api-access-l9dtj" pod "3963bd73-a6f8-4650-a9ef-8ffd1a2ee63f" (UID: "3963bd73-a6f8-4650-a9ef-8ffd1a2ee63f"). InnerVolumeSpecName "kube-api-access-l9dtj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:42:29.217757 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:29.217734    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3963bd73-a6f8-4650-a9ef-8ffd1a2ee63f-util" pod "3963bd73-a6f8-4650-a9ef-8ffd1a2ee63f" (UID: "3963bd73-a6f8-4650-a9ef-8ffd1a2ee63f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:42:29.219090 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:29.219066    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54d00e9e-313d-4fde-87a2-e641c4caf568-util" pod "54d00e9e-313d-4fde-87a2-e641c4caf568" (UID: "54d00e9e-313d-4fde-87a2-e641c4caf568"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:42:29.311567 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:29.311517    4521 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3963bd73-a6f8-4650-a9ef-8ffd1a2ee63f-bundle\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:42:29.311567 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:29.311560    4521 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3963bd73-a6f8-4650-a9ef-8ffd1a2ee63f-util\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:42:29.311567 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:29.311571    4521 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnhh8\" (UniqueName: \"kubernetes.io/projected/54d00e9e-313d-4fde-87a2-e641c4caf568-kube-api-access-dnhh8\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:42:29.311567 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:29.311583    4521 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/54d00e9e-313d-4fde-87a2-e641c4caf568-util\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:42:29.311836 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:29.311593    4521 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/54d00e9e-313d-4fde-87a2-e641c4caf568-bundle\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:42:29.311836 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:29.311601    4521 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9dtj\" (UniqueName: \"kubernetes.io/projected/3963bd73-a6f8-4650-a9ef-8ffd1a2ee63f-kube-api-access-l9dtj\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:42:29.971503 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:29.971462    4521 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5d46577d6f-rvfc7"]
Oct 30 20:42:29.971803 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:29.971784    4521 cpu_manager.go:414] "containerMap: removing container" podUID="824a2b93-8ec6-4fe1-9d0d-abf8cc679abf" containerName="pull"
Oct 30 20:42:29.971803 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:29.971797    4521 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="824a2b93-8ec6-4fe1-9d0d-abf8cc679abf" containerName="pull"
Oct 30 20:42:29.971884 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:29.971809    4521 cpu_manager.go:414] "containerMap: removing container" podUID="747d9d7a-fdca-4839-afe2-ef5933b3cc16" containerName="extract"
Oct 30 20:42:29.971884 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:29.971814    4521 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="747d9d7a-fdca-4839-afe2-ef5933b3cc16" containerName="extract"
Oct 30 20:42:29.971884 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:29.971822    4521 cpu_manager.go:414] "containerMap: removing container" podUID="62d1efb9-0cea-45ca-a468-31f81124bf4b" containerName="extract"
Oct 30 20:42:29.971884 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:29.971827    4521 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62d1efb9-0cea-45ca-a468-31f81124bf4b" containerName="extract"
Oct 30 20:42:29.971884 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:29.971836    4521 cpu_manager.go:414] "containerMap: removing container" podUID="54d00e9e-313d-4fde-87a2-e641c4caf568" containerName="pull"
Oct 30 20:42:29.971884 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:29.971841    4521 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54d00e9e-313d-4fde-87a2-e641c4caf568" containerName="pull"
Oct 30 20:42:29.971884 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:29.971847    4521 cpu_manager.go:414] "containerMap: removing container" podUID="de36102c-fee1-40dd-ba3f-95691343bae0" containerName="extract"
Oct 30 20:42:29.971884 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:29.971852    4521 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de36102c-fee1-40dd-ba3f-95691343bae0" containerName="extract"
Oct 30 20:42:29.971884 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:29.971858    4521 cpu_manager.go:414] "containerMap: removing container" podUID="1e8b2d49-82b3-47fa-8317-a5df38e00cb4" containerName="util"
Oct 30 20:42:29.971884 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:29.971862    4521 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e8b2d49-82b3-47fa-8317-a5df38e00cb4" containerName="util"
Oct 30 20:42:29.971884 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:29.971869    4521 cpu_manager.go:414] "containerMap: removing container" podUID="be269a25-d606-4ec1-acb4-38eedf4f201c" containerName="pull"
Oct 30 20:42:29.971884 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:29.971874    4521 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be269a25-d606-4ec1-acb4-38eedf4f201c" containerName="pull"
Oct 30 20:42:29.971884 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:29.971882    4521 cpu_manager.go:414] "containerMap: removing container" podUID="fd75cc69-c4d2-4e9f-9215-f0e0b81bd64e" containerName="util"
Oct 30 20:42:29.971884 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:29.971888    4521 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd75cc69-c4d2-4e9f-9215-f0e0b81bd64e" containerName="util"
Oct 30 20:42:29.972314 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:29.971898    4521 cpu_manager.go:414] "containerMap: removing container" podUID="1e8b2d49-82b3-47fa-8317-a5df38e00cb4" containerName="pull"
Oct 30 20:42:29.972314 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:29.971903    4521 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e8b2d49-82b3-47fa-8317-a5df38e00cb4" containerName="pull"
Oct 30 20:42:29.972314 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:29.971909    4521 cpu_manager.go:414] "containerMap: removing container" podUID="62d1efb9-0cea-45ca-a468-31f81124bf4b" containerName="pull"
Oct 30 20:42:29.972314 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:29.971913    4521 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62d1efb9-0cea-45ca-a468-31f81124bf4b" containerName="pull"
Oct 30 20:42:29.972314 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:29.971921    4521 cpu_manager.go:414] "containerMap: removing container" podUID="1e8b2d49-82b3-47fa-8317-a5df38e00cb4" containerName="extract"
Oct 30 20:42:29.972314 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:29.971925    4521 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e8b2d49-82b3-47fa-8317-a5df38e00cb4" containerName="extract"
Oct 30 20:42:29.972314 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:29.971933    4521 cpu_manager.go:414] "containerMap: removing container" podUID="62d1efb9-0cea-45ca-a468-31f81124bf4b" containerName="util"
Oct 30 20:42:29.972314 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:29.971937    4521 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62d1efb9-0cea-45ca-a468-31f81124bf4b" containerName="util"
Oct 30 20:42:29.972314 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:29.971944    4521 cpu_manager.go:414] "containerMap: removing container" podUID="3963bd73-a6f8-4650-a9ef-8ffd1a2ee63f" containerName="util"
Oct 30 20:42:29.972314 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:29.971950    4521 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3963bd73-a6f8-4650-a9ef-8ffd1a2ee63f" containerName="util"
Oct 30 20:42:29.972314 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:29.971959    4521 cpu_manager.go:414] "containerMap: removing container" podUID="de36102c-fee1-40dd-ba3f-95691343bae0" containerName="pull"
Oct 30 20:42:29.972314 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:29.971963    4521 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de36102c-fee1-40dd-ba3f-95691343bae0" containerName="pull"
Oct 30 20:42:29.972314 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:29.971969    4521 cpu_manager.go:414] "containerMap: removing container" podUID="be269a25-d606-4ec1-acb4-38eedf4f201c" containerName="extract"
Oct 30 20:42:29.972314 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:29.971973    4521 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be269a25-d606-4ec1-acb4-38eedf4f201c" containerName="extract"
Oct 30 20:42:29.972314 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:29.971982    4521 cpu_manager.go:414] "containerMap: removing container" podUID="3963bd73-a6f8-4650-a9ef-8ffd1a2ee63f" containerName="extract"
Oct 30 20:42:29.972314 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:29.971987    4521 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3963bd73-a6f8-4650-a9ef-8ffd1a2ee63f" containerName="extract"
Oct 30 20:42:29.972314 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:29.971992    4521 cpu_manager.go:414] "containerMap: removing container" podUID="de36102c-fee1-40dd-ba3f-95691343bae0" containerName="util"
Oct 30 20:42:29.972314 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:29.971996    4521 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de36102c-fee1-40dd-ba3f-95691343bae0" containerName="util"
Oct 30 20:42:29.972314 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:29.972003    4521 cpu_manager.go:414] "containerMap: removing container" podUID="824a2b93-8ec6-4fe1-9d0d-abf8cc679abf" containerName="util"
Oct 30 20:42:29.972314 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:29.972007    4521 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="824a2b93-8ec6-4fe1-9d0d-abf8cc679abf" containerName="util"
Oct 30 20:42:29.972314 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:29.972012    4521 cpu_manager.go:414] "containerMap: removing container" podUID="824a2b93-8ec6-4fe1-9d0d-abf8cc679abf" containerName="extract"
Oct 30 20:42:29.972314 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:29.972017    4521 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="824a2b93-8ec6-4fe1-9d0d-abf8cc679abf" containerName="extract"
Oct 30 20:42:29.972314 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:29.972025    4521 cpu_manager.go:414] "containerMap: removing container" podUID="fd75cc69-c4d2-4e9f-9215-f0e0b81bd64e" containerName="extract"
Oct 30 20:42:29.972314 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:29.972029    4521 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd75cc69-c4d2-4e9f-9215-f0e0b81bd64e" containerName="extract"
Oct 30 20:42:29.972314 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:29.972035    4521 cpu_manager.go:414] "containerMap: removing container" podUID="be269a25-d606-4ec1-acb4-38eedf4f201c" containerName="util"
Oct 30 20:42:29.972314 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:29.972040    4521 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be269a25-d606-4ec1-acb4-38eedf4f201c" containerName="util"
Oct 30 20:42:29.972314 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:29.972045    4521 cpu_manager.go:414] "containerMap: removing container" podUID="747d9d7a-fdca-4839-afe2-ef5933b3cc16" containerName="util"
Oct 30 20:42:29.972314 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:29.972050    4521 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="747d9d7a-fdca-4839-afe2-ef5933b3cc16" containerName="util"
Oct 30 20:42:29.972314 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:29.972055    4521 cpu_manager.go:414] "containerMap: removing container" podUID="747d9d7a-fdca-4839-afe2-ef5933b3cc16" containerName="pull"
Oct 30 20:42:29.972314 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:29.972059    4521 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="747d9d7a-fdca-4839-afe2-ef5933b3cc16" containerName="pull"
Oct 30 20:42:29.972314 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:29.972065    4521 cpu_manager.go:414] "containerMap: removing container" podUID="54d00e9e-313d-4fde-87a2-e641c4caf568" containerName="extract"
Oct 30 20:42:29.972314 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:29.972069    4521 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54d00e9e-313d-4fde-87a2-e641c4caf568" containerName="extract"
Oct 30 20:42:29.972314 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:29.972076    4521 cpu_manager.go:414] "containerMap: removing container" podUID="3963bd73-a6f8-4650-a9ef-8ffd1a2ee63f" containerName="pull"
Oct 30 20:42:29.972314 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:29.972080    4521 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3963bd73-a6f8-4650-a9ef-8ffd1a2ee63f" containerName="pull"
Oct 30 20:42:29.972314 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:29.972086    4521 cpu_manager.go:414] "containerMap: removing container" podUID="54d00e9e-313d-4fde-87a2-e641c4caf568" containerName="util"
Oct 30 20:42:29.972314 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:29.972090    4521 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54d00e9e-313d-4fde-87a2-e641c4caf568" containerName="util"
Oct 30 20:42:29.972314 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:29.972096    4521 cpu_manager.go:414] "containerMap: removing container" podUID="fd75cc69-c4d2-4e9f-9215-f0e0b81bd64e" containerName="pull"
Oct 30 20:42:29.972314 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:29.972100    4521 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd75cc69-c4d2-4e9f-9215-f0e0b81bd64e" containerName="pull"
Oct 30 20:42:29.972314 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:29.972141    4521 memory_manager.go:365] "RemoveStaleState removing state" podUID="3963bd73-a6f8-4650-a9ef-8ffd1a2ee63f" containerName="extract"
Oct 30 20:42:29.972314 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:29.972147    4521 memory_manager.go:365] "RemoveStaleState removing state" podUID="824a2b93-8ec6-4fe1-9d0d-abf8cc679abf" containerName="extract"
Oct 30 20:42:29.972314 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:29.972154    4521 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd75cc69-c4d2-4e9f-9215-f0e0b81bd64e" containerName="extract"
Oct 30 20:42:29.972314 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:29.972159    4521 memory_manager.go:365] "RemoveStaleState removing state" podUID="54d00e9e-313d-4fde-87a2-e641c4caf568" containerName="extract"
Oct 30 20:42:29.972314 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:29.972165    4521 memory_manager.go:365] "RemoveStaleState removing state" podUID="747d9d7a-fdca-4839-afe2-ef5933b3cc16" containerName="extract"
Oct 30 20:42:29.972314 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:29.972170    4521 memory_manager.go:365] "RemoveStaleState removing state" podUID="62d1efb9-0cea-45ca-a468-31f81124bf4b" containerName="extract"
Oct 30 20:42:29.972314 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:29.972175    4521 memory_manager.go:365] "RemoveStaleState removing state" podUID="de36102c-fee1-40dd-ba3f-95691343bae0" containerName="extract"
Oct 30 20:42:29.972314 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:29.972181    4521 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e8b2d49-82b3-47fa-8317-a5df38e00cb4" containerName="extract"
Oct 30 20:42:29.972314 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:29.972187    4521 memory_manager.go:365] "RemoveStaleState removing state" podUID="be269a25-d606-4ec1-acb4-38eedf4f201c" containerName="extract"
Oct 30 20:42:29.974609 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:29.974592    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5d46577d6f-rvfc7"
Oct 30 20:42:29.981815 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:29.981793    4521 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Oct 30 20:42:29.981979 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:29.981961    4521 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Oct 30 20:42:29.982050 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:29.982038    4521 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Oct 30 20:42:29.983427 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:29.983410    4521 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-hvr5z\""
Oct 30 20:42:29.983493 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:29.983442    4521 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Oct 30 20:42:29.983531 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:29.983505    4521 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Oct 30 20:42:30.017068 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:30.017029    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f09f227c-2059-4bb6-af30-f18c4c261487-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5d46577d6f-rvfc7\" (UID: \"f09f227c-2059-4bb6-af30-f18c4c261487\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5d46577d6f-rvfc7"
Oct 30 20:42:30.017068 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:30.017071    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/f09f227c-2059-4bb6-af30-f18c4c261487-metrics-cert\") pod \"opendatahub-operator-controller-manager-5d46577d6f-rvfc7\" (UID: \"f09f227c-2059-4bb6-af30-f18c4c261487\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5d46577d6f-rvfc7"
Oct 30 20:42:30.017247 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:30.017189    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f09f227c-2059-4bb6-af30-f18c4c261487-tmp\") pod \"opendatahub-operator-controller-manager-5d46577d6f-rvfc7\" (UID: \"f09f227c-2059-4bb6-af30-f18c4c261487\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5d46577d6f-rvfc7"
Oct 30 20:42:30.017247 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:30.017235    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f09f227c-2059-4bb6-af30-f18c4c261487-webhook-cert\") pod \"opendatahub-operator-controller-manager-5d46577d6f-rvfc7\" (UID: \"f09f227c-2059-4bb6-af30-f18c4c261487\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5d46577d6f-rvfc7"
Oct 30 20:42:30.017318 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:30.017295    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f09f227c-2059-4bb6-af30-f18c4c261487-manifests\") pod \"opendatahub-operator-controller-manager-5d46577d6f-rvfc7\" (UID: \"f09f227c-2059-4bb6-af30-f18c4c261487\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5d46577d6f-rvfc7"
Oct 30 20:42:30.017384 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:30.017368    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvxjz\" (UniqueName: \"kubernetes.io/projected/f09f227c-2059-4bb6-af30-f18c4c261487-kube-api-access-bvxjz\") pod \"opendatahub-operator-controller-manager-5d46577d6f-rvfc7\" (UID: \"f09f227c-2059-4bb6-af30-f18c4c261487\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5d46577d6f-rvfc7"
Oct 30 20:42:30.028360 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:30.028304    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5d46577d6f-rvfc7"]
Oct 30 20:42:30.032573 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:30.032552    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cd4bdk"
Oct 30 20:42:30.032573 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:30.032558    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cd4bdk" event={"ID":"54d00e9e-313d-4fde-87a2-e641c4caf568","Type":"ContainerDied","Data":"b50ea07ab6529e58675b4b0fdef7dca5f2cbc900e7f6e20d788bd447b57b5369"}
Oct 30 20:42:30.032757 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:30.032607    4521 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b50ea07ab6529e58675b4b0fdef7dca5f2cbc900e7f6e20d788bd447b57b5369"
Oct 30 20:42:30.034672 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:30.034647    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cpdbc"
Oct 30 20:42:30.034672 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:30.034650    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cpdbc" event={"ID":"3963bd73-a6f8-4650-a9ef-8ffd1a2ee63f","Type":"ContainerDied","Data":"0af16f3d3feadda1dea1e39cf3ec0ff87a01ed1869bf623c377d60a8dd6bffee"}
Oct 30 20:42:30.034843 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:30.034694    4521 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0af16f3d3feadda1dea1e39cf3ec0ff87a01ed1869bf623c377d60a8dd6bffee"
Oct 30 20:42:30.118112 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:30.118077    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f09f227c-2059-4bb6-af30-f18c4c261487-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5d46577d6f-rvfc7\" (UID: \"f09f227c-2059-4bb6-af30-f18c4c261487\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5d46577d6f-rvfc7"
Oct 30 20:42:30.118112 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:30.118110    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/f09f227c-2059-4bb6-af30-f18c4c261487-metrics-cert\") pod \"opendatahub-operator-controller-manager-5d46577d6f-rvfc7\" (UID: \"f09f227c-2059-4bb6-af30-f18c4c261487\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5d46577d6f-rvfc7"
Oct 30 20:42:30.118361 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:30.118148    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f09f227c-2059-4bb6-af30-f18c4c261487-tmp\") pod \"opendatahub-operator-controller-manager-5d46577d6f-rvfc7\" (UID: \"f09f227c-2059-4bb6-af30-f18c4c261487\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5d46577d6f-rvfc7"
Oct 30 20:42:30.118361 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:30.118178    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f09f227c-2059-4bb6-af30-f18c4c261487-webhook-cert\") pod \"opendatahub-operator-controller-manager-5d46577d6f-rvfc7\" (UID: \"f09f227c-2059-4bb6-af30-f18c4c261487\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5d46577d6f-rvfc7"
Oct 30 20:42:30.118361 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:30.118194    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f09f227c-2059-4bb6-af30-f18c4c261487-manifests\") pod \"opendatahub-operator-controller-manager-5d46577d6f-rvfc7\" (UID: \"f09f227c-2059-4bb6-af30-f18c4c261487\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5d46577d6f-rvfc7"
Oct 30 20:42:30.118361 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:30.118221    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvxjz\" (UniqueName: \"kubernetes.io/projected/f09f227c-2059-4bb6-af30-f18c4c261487-kube-api-access-bvxjz\") pod \"opendatahub-operator-controller-manager-5d46577d6f-rvfc7\" (UID: \"f09f227c-2059-4bb6-af30-f18c4c261487\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5d46577d6f-rvfc7"
Oct 30 20:42:30.118758 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:30.118728    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f09f227c-2059-4bb6-af30-f18c4c261487-manifests\") pod \"opendatahub-operator-controller-manager-5d46577d6f-rvfc7\" (UID: \"f09f227c-2059-4bb6-af30-f18c4c261487\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5d46577d6f-rvfc7"
Oct 30 20:42:30.120505 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:30.120483    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f09f227c-2059-4bb6-af30-f18c4c261487-tmp\") pod \"opendatahub-operator-controller-manager-5d46577d6f-rvfc7\" (UID: \"f09f227c-2059-4bb6-af30-f18c4c261487\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5d46577d6f-rvfc7"
Oct 30 20:42:30.120744 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:30.120727    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f09f227c-2059-4bb6-af30-f18c4c261487-webhook-cert\") pod \"opendatahub-operator-controller-manager-5d46577d6f-rvfc7\" (UID: \"f09f227c-2059-4bb6-af30-f18c4c261487\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5d46577d6f-rvfc7"
Oct 30 20:42:30.120782 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:30.120732    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f09f227c-2059-4bb6-af30-f18c4c261487-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5d46577d6f-rvfc7\" (UID: \"f09f227c-2059-4bb6-af30-f18c4c261487\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5d46577d6f-rvfc7"
Oct 30 20:42:30.120782 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:30.120767    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/f09f227c-2059-4bb6-af30-f18c4c261487-metrics-cert\") pod \"opendatahub-operator-controller-manager-5d46577d6f-rvfc7\" (UID: \"f09f227c-2059-4bb6-af30-f18c4c261487\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5d46577d6f-rvfc7"
Oct 30 20:42:30.131542 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:30.131518    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvxjz\" (UniqueName: \"kubernetes.io/projected/f09f227c-2059-4bb6-af30-f18c4c261487-kube-api-access-bvxjz\") pod \"opendatahub-operator-controller-manager-5d46577d6f-rvfc7\" (UID: \"f09f227c-2059-4bb6-af30-f18c4c261487\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5d46577d6f-rvfc7"
Oct 30 20:42:30.287133 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:30.287042    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5d46577d6f-rvfc7"
Oct 30 20:42:30.591701 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:30.591674    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5d46577d6f-rvfc7"]
Oct 30 20:42:30.592282 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:42:30.592257    4521 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf09f227c_2059_4bb6_af30_f18c4c261487.slice/crio-9626b3d3d0d69ecdb6c98fa252f081bd5057597e392003fe57ce64397e102092 WatchSource:0}: Error finding container 9626b3d3d0d69ecdb6c98fa252f081bd5057597e392003fe57ce64397e102092: Status 404 returned error can't find the container with id 9626b3d3d0d69ecdb6c98fa252f081bd5057597e392003fe57ce64397e102092
Oct 30 20:42:31.039587 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:31.039554    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5d46577d6f-rvfc7" event={"ID":"f09f227c-2059-4bb6-af30-f18c4c261487","Type":"ContainerStarted","Data":"9626b3d3d0d69ecdb6c98fa252f081bd5057597e392003fe57ce64397e102092"}
Oct 30 20:42:34.056812 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:34.056758    4521 generic.go:356] "Generic (PLEG): container finished" podID="f09f227c-2059-4bb6-af30-f18c4c261487" containerID="adddebfaabc7592919d5279e19d68df23b2a1f9762b69a8de61985c7dc44c2df" exitCode=0
Oct 30 20:42:34.057297 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:34.056867    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5d46577d6f-rvfc7" event={"ID":"f09f227c-2059-4bb6-af30-f18c4c261487","Type":"ContainerDied","Data":"adddebfaabc7592919d5279e19d68df23b2a1f9762b69a8de61985c7dc44c2df"}
Oct 30 20:42:35.063824 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:35.063788    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5d46577d6f-rvfc7" event={"ID":"f09f227c-2059-4bb6-af30-f18c4c261487","Type":"ContainerStarted","Data":"92ff2bddd382b84a250d559dad2e87178a5d9112af4a967da1d4e89ec12a088b"}
Oct 30 20:42:35.064216 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:35.063900    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5d46577d6f-rvfc7"
Oct 30 20:42:35.130499 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:35.130431    4521 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5d46577d6f-rvfc7" podStartSLOduration=2.48426668 podStartE2EDuration="6.130410066s" totalImagesPullingTime="3.156805943s" totalInitContainerRuntime="489.337443ms" isStatefulPod=false podCreationTimestamp="2026-10-30 20:42:29 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:42:35.127980843 +0000 UTC m=+222.473602115" watchObservedRunningTime="2026-10-30 20:42:35.130410066 +0000 UTC m=+222.476031305"
Oct 30 20:42:37.893800 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:37.893766    4521 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-ws2wn"]
Oct 30 20:42:37.896758 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:37.896740    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-ws2wn"
Oct 30 20:42:37.907219 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:37.907185    4521 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Oct 30 20:42:37.907318 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:37.907296    4521 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Oct 30 20:42:37.907386 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:37.907336    4521 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-qwblw\""
Oct 30 20:42:37.914250 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:37.914218    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-ws2wn"]
Oct 30 20:42:37.979401 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:37.979355    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j76vf\" (UniqueName: \"kubernetes.io/projected/8982dfae-c4f0-4b25-82fa-bb25f015bfa0-kube-api-access-j76vf\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-ws2wn\" (UID: \"8982dfae-c4f0-4b25-82fa-bb25f015bfa0\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-ws2wn"
Oct 30 20:42:37.979672 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:37.979653    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8982dfae-c4f0-4b25-82fa-bb25f015bfa0-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-ws2wn\" (UID: \"8982dfae-c4f0-4b25-82fa-bb25f015bfa0\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-ws2wn"
Oct 30 20:42:38.080786 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:38.080749    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j76vf\" (UniqueName: \"kubernetes.io/projected/8982dfae-c4f0-4b25-82fa-bb25f015bfa0-kube-api-access-j76vf\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-ws2wn\" (UID: \"8982dfae-c4f0-4b25-82fa-bb25f015bfa0\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-ws2wn"
Oct 30 20:42:38.080969 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:38.080799    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8982dfae-c4f0-4b25-82fa-bb25f015bfa0-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-ws2wn\" (UID: \"8982dfae-c4f0-4b25-82fa-bb25f015bfa0\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-ws2wn"
Oct 30 20:42:38.081157 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:38.081142    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8982dfae-c4f0-4b25-82fa-bb25f015bfa0-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-ws2wn\" (UID: \"8982dfae-c4f0-4b25-82fa-bb25f015bfa0\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-ws2wn"
Oct 30 20:42:38.094579 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:38.094546    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j76vf\" (UniqueName: \"kubernetes.io/projected/8982dfae-c4f0-4b25-82fa-bb25f015bfa0-kube-api-access-j76vf\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-ws2wn\" (UID: \"8982dfae-c4f0-4b25-82fa-bb25f015bfa0\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-ws2wn"
Oct 30 20:42:38.207672 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:38.207639    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-ws2wn"
Oct 30 20:42:38.531536 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:38.531512    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-ws2wn"]
Oct 30 20:42:38.532646 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:42:38.532615    4521 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8982dfae_c4f0_4b25_82fa_bb25f015bfa0.slice/crio-3a94d211126ddd1dca3b06a299c5620bb10c41194c97fc9ec8b0bfc9977f7695 WatchSource:0}: Error finding container 3a94d211126ddd1dca3b06a299c5620bb10c41194c97fc9ec8b0bfc9977f7695: Status 404 returned error can't find the container with id 3a94d211126ddd1dca3b06a299c5620bb10c41194c97fc9ec8b0bfc9977f7695
Oct 30 20:42:39.081937 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:39.081901    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-ws2wn" event={"ID":"8982dfae-c4f0-4b25-82fa-bb25f015bfa0","Type":"ContainerStarted","Data":"3a94d211126ddd1dca3b06a299c5620bb10c41194c97fc9ec8b0bfc9977f7695"}
Oct 30 20:42:41.094745 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:41.094708    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-ws2wn" event={"ID":"8982dfae-c4f0-4b25-82fa-bb25f015bfa0","Type":"ContainerStarted","Data":"c112346ca061a7869ab9921ebbab3d05c9ff2e46a314a826afe647b87ebcc77a"}
Oct 30 20:42:41.125028 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:41.124974    4521 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-ws2wn" podStartSLOduration=2.172992754 podStartE2EDuration="4.124959274s" totalImagesPullingTime="1.95196652s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:42:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:42:41.12205069 +0000 UTC m=+228.467671927" watchObservedRunningTime="2026-10-30 20:42:41.124959274 +0000 UTC m=+228.470580508"
Oct 30 20:42:45.314871 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:45.314836    4521 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-cjgfs"]
Oct 30 20:42:45.317962 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:45.317944    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-cjgfs"
Oct 30 20:42:45.339719 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:45.339688    4521 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Oct 30 20:42:45.341243 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:45.341226    4521 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Oct 30 20:42:45.348509 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:45.348483    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-cjgfs"]
Oct 30 20:42:45.352906 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:45.352888    4521 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-klw8k\""
Oct 30 20:42:45.444239 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:45.444198    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f7a784af-42db-4056-8466-81aff3558221-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-cjgfs\" (UID: \"f7a784af-42db-4056-8466-81aff3558221\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-cjgfs"
Oct 30 20:42:45.444239 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:45.444237    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfbbm\" (UniqueName: \"kubernetes.io/projected/f7a784af-42db-4056-8466-81aff3558221-kube-api-access-rfbbm\") pod \"cert-manager-cainjector-5c57bbd4fc-cjgfs\" (UID: \"f7a784af-42db-4056-8466-81aff3558221\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-cjgfs"
Oct 30 20:42:45.544924 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:45.544893    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f7a784af-42db-4056-8466-81aff3558221-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-cjgfs\" (UID: \"f7a784af-42db-4056-8466-81aff3558221\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-cjgfs"
Oct 30 20:42:45.545081 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:45.544928    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfbbm\" (UniqueName: \"kubernetes.io/projected/f7a784af-42db-4056-8466-81aff3558221-kube-api-access-rfbbm\") pod \"cert-manager-cainjector-5c57bbd4fc-cjgfs\" (UID: \"f7a784af-42db-4056-8466-81aff3558221\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-cjgfs"
Oct 30 20:42:45.566528 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:45.566443    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f7a784af-42db-4056-8466-81aff3558221-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-cjgfs\" (UID: \"f7a784af-42db-4056-8466-81aff3558221\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-cjgfs"
Oct 30 20:42:45.567913 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:45.567891    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfbbm\" (UniqueName: \"kubernetes.io/projected/f7a784af-42db-4056-8466-81aff3558221-kube-api-access-rfbbm\") pod \"cert-manager-cainjector-5c57bbd4fc-cjgfs\" (UID: \"f7a784af-42db-4056-8466-81aff3558221\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-cjgfs"
Oct 30 20:42:45.642942 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:45.642891    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-cjgfs"
Oct 30 20:42:45.925500 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:45.925418    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-cjgfs"]
Oct 30 20:42:45.928740 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:42:45.928713    4521 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf7a784af_42db_4056_8466_81aff3558221.slice/crio-071597ba980a5dd53fca2eaa39b6b1f5b81118ac1d7cc047245d6c799c1144fb WatchSource:0}: Error finding container 071597ba980a5dd53fca2eaa39b6b1f5b81118ac1d7cc047245d6c799c1144fb: Status 404 returned error can't find the container with id 071597ba980a5dd53fca2eaa39b6b1f5b81118ac1d7cc047245d6c799c1144fb
Oct 30 20:42:46.070932 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:46.070895    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5d46577d6f-rvfc7"
Oct 30 20:42:46.118766 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:46.118732    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-cjgfs" event={"ID":"f7a784af-42db-4056-8466-81aff3558221","Type":"ContainerStarted","Data":"071597ba980a5dd53fca2eaa39b6b1f5b81118ac1d7cc047245d6c799c1144fb"}
Oct 30 20:42:46.428468 ip-10-0-131-125 kubenswrapper[4521]: E1030 20:42:46.428433    4521 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lkn2k" containerName="extractor"
Oct 30 20:42:49.135011 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:49.134979    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-cjgfs" event={"ID":"f7a784af-42db-4056-8466-81aff3558221","Type":"ContainerStarted","Data":"0794b653641ffe7625b1005c4540c80605bafd24dab84764cdc33fc4dae0491f"}
Oct 30 20:42:50.813194 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:50.813121    4521 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-cjgfs" podStartSLOduration=3.114567636 podStartE2EDuration="5.813104697s" totalImagesPullingTime="2.698537061s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:42:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:42:49.162124039 +0000 UTC m=+236.507745278" watchObservedRunningTime="2026-10-30 20:42:50.813104697 +0000 UTC m=+238.158725940"
Oct 30 20:42:50.815184 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:50.815153    4521 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-x9bck"]
Oct 30 20:42:50.817264 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:50.817244    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-x9bck"
Oct 30 20:42:50.833572 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:50.833547    4521 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Oct 30 20:42:50.834068 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:50.834052    4521 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Oct 30 20:42:50.842747 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:50.842733    4521 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-2bcgw\""
Oct 30 20:42:50.846783 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:50.846757    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-x9bck"]
Oct 30 20:42:50.894426 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:50.894397    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78qxs\" (UniqueName: \"kubernetes.io/projected/8235f71b-607f-4cd7-a69a-de5b52fa90cd-kube-api-access-78qxs\") pod \"authorino-operator-7b7bb4bc77-x9bck\" (UID: \"8235f71b-607f-4cd7-a69a-de5b52fa90cd\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-x9bck"
Oct 30 20:42:50.995250 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:50.995214    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78qxs\" (UniqueName: \"kubernetes.io/projected/8235f71b-607f-4cd7-a69a-de5b52fa90cd-kube-api-access-78qxs\") pod \"authorino-operator-7b7bb4bc77-x9bck\" (UID: \"8235f71b-607f-4cd7-a69a-de5b52fa90cd\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-x9bck"
Oct 30 20:42:51.007665 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:51.007631    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78qxs\" (UniqueName: \"kubernetes.io/projected/8235f71b-607f-4cd7-a69a-de5b52fa90cd-kube-api-access-78qxs\") pod \"authorino-operator-7b7bb4bc77-x9bck\" (UID: \"8235f71b-607f-4cd7-a69a-de5b52fa90cd\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-x9bck"
Oct 30 20:42:51.128643 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:51.128558    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-x9bck"
Oct 30 20:42:51.409561 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:51.409536    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-x9bck"]
Oct 30 20:42:51.412035 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:42:51.412003    4521 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8235f71b_607f_4cd7_a69a_de5b52fa90cd.slice/crio-eb9f89b0a29b602b9bccd0ebecee1f520320a0e5f08651b15ec176d7b8629321 WatchSource:0}: Error finding container eb9f89b0a29b602b9bccd0ebecee1f520320a0e5f08651b15ec176d7b8629321: Status 404 returned error can't find the container with id eb9f89b0a29b602b9bccd0ebecee1f520320a0e5f08651b15ec176d7b8629321
Oct 30 20:42:52.150168 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:52.150122    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-x9bck" event={"ID":"8235f71b-607f-4cd7-a69a-de5b52fa90cd","Type":"ContainerStarted","Data":"eb9f89b0a29b602b9bccd0ebecee1f520320a0e5f08651b15ec176d7b8629321"}
Oct 30 20:42:53.764406 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:53.764372    4521 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-f97mf"]
Oct 30 20:42:53.771929 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:53.771902    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-f97mf"
Oct 30 20:42:53.782079 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:53.782043    4521 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Oct 30 20:42:53.786032 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:53.786010    4521 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-prr9f\""
Oct 30 20:42:53.800335 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:53.800284    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-f97mf"]
Oct 30 20:42:53.919923 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:53.919887    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vl4q\" (UniqueName: \"kubernetes.io/projected/a78e2ba4-f4b7-4df7-9e8e-e7cf0e717c42-kube-api-access-7vl4q\") pod \"dns-operator-controller-manager-8557cbdb84-f97mf\" (UID: \"a78e2ba4-f4b7-4df7-9e8e-e7cf0e717c42\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-f97mf"
Oct 30 20:42:54.020646 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:54.020563    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vl4q\" (UniqueName: \"kubernetes.io/projected/a78e2ba4-f4b7-4df7-9e8e-e7cf0e717c42-kube-api-access-7vl4q\") pod \"dns-operator-controller-manager-8557cbdb84-f97mf\" (UID: \"a78e2ba4-f4b7-4df7-9e8e-e7cf0e717c42\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-f97mf"
Oct 30 20:42:54.041974 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:54.041940    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vl4q\" (UniqueName: \"kubernetes.io/projected/a78e2ba4-f4b7-4df7-9e8e-e7cf0e717c42-kube-api-access-7vl4q\") pod \"dns-operator-controller-manager-8557cbdb84-f97mf\" (UID: \"a78e2ba4-f4b7-4df7-9e8e-e7cf0e717c42\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-f97mf"
Oct 30 20:42:54.089375 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:54.089340    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-f97mf"
Oct 30 20:42:54.392096 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:54.392068    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-f97mf"]
Oct 30 20:42:54.424060 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:42:54.424020    4521 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda78e2ba4_f4b7_4df7_9e8e_e7cf0e717c42.slice/crio-ef9447ba7751e10284f73b58e87bfe6c4d286baa3121f9af3da7b56ddfbb3820 WatchSource:0}: Error finding container ef9447ba7751e10284f73b58e87bfe6c4d286baa3121f9af3da7b56ddfbb3820: Status 404 returned error can't find the container with id ef9447ba7751e10284f73b58e87bfe6c4d286baa3121f9af3da7b56ddfbb3820
Oct 30 20:42:55.171253 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:55.171216    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-f97mf" event={"ID":"a78e2ba4-f4b7-4df7-9e8e-e7cf0e717c42","Type":"ContainerStarted","Data":"ef9447ba7751e10284f73b58e87bfe6c4d286baa3121f9af3da7b56ddfbb3820"}
Oct 30 20:42:56.178399 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:56.178360    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-x9bck" event={"ID":"8235f71b-607f-4cd7-a69a-de5b52fa90cd","Type":"ContainerStarted","Data":"0538781be002de03384831fcb35f1403264e1160ad5cfaf8f6c7408478fe2b60"}
Oct 30 20:42:56.178840 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:56.178531    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-x9bck"
Oct 30 20:42:56.220836 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:56.220660    4521 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7b7bb4bc77-x9bck" podStartSLOduration=2.415477022 podStartE2EDuration="6.220643867s" totalImagesPullingTime="3.805166845s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:42:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:42:56.218143113 +0000 UTC m=+243.563764352" watchObservedRunningTime="2026-10-30 20:42:56.220643867 +0000 UTC m=+243.566265104"
Oct 30 20:42:57.184567 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:57.184518    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-f97mf" event={"ID":"a78e2ba4-f4b7-4df7-9e8e-e7cf0e717c42","Type":"ContainerStarted","Data":"ecedf887bcf05181ae49315a74526c10eadfb6d90fffed85be58c66a04313db2"}
Oct 30 20:42:57.185054 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:57.184735    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-f97mf"
Oct 30 20:42:57.213240 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:57.213194    4521 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-f97mf" podStartSLOduration=2.4735782459999998 podStartE2EDuration="4.213181535s" totalImagesPullingTime="1.739603289s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:42:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:42:57.212004959 +0000 UTC m=+244.557626197" watchObservedRunningTime="2026-10-30 20:42:57.213181535 +0000 UTC m=+244.558802773"
Oct 30 20:42:59.953258 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:59.953221    4521 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-szfml"]
Oct 30 20:42:59.955790 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:59.955774    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-szfml"
Oct 30 20:42:59.973198 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:59.973169    4521 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-lpzsx\""
Oct 30 20:42:59.974955 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:42:59.974931    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-szfml"]
Oct 30 20:43:00.081438 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:43:00.076948    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9e9617c7-d6d3-447b-8dc2-a0218e3f6a14-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-szfml\" (UID: \"9e9617c7-d6d3-447b-8dc2-a0218e3f6a14\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-szfml"
Oct 30 20:43:00.081438 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:43:00.077054    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjm8w\" (UniqueName: \"kubernetes.io/projected/9e9617c7-d6d3-447b-8dc2-a0218e3f6a14-kube-api-access-gjm8w\") pod \"kuadrant-operator-controller-manager-678d87766f-szfml\" (UID: \"9e9617c7-d6d3-447b-8dc2-a0218e3f6a14\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-szfml"
Oct 30 20:43:00.178289 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:43:00.178247    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9e9617c7-d6d3-447b-8dc2-a0218e3f6a14-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-szfml\" (UID: \"9e9617c7-d6d3-447b-8dc2-a0218e3f6a14\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-szfml"
Oct 30 20:43:00.178482 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:43:00.178304    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjm8w\" (UniqueName: \"kubernetes.io/projected/9e9617c7-d6d3-447b-8dc2-a0218e3f6a14-kube-api-access-gjm8w\") pod \"kuadrant-operator-controller-manager-678d87766f-szfml\" (UID: \"9e9617c7-d6d3-447b-8dc2-a0218e3f6a14\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-szfml"
Oct 30 20:43:00.178669 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:43:00.178646    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9e9617c7-d6d3-447b-8dc2-a0218e3f6a14-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-szfml\" (UID: \"9e9617c7-d6d3-447b-8dc2-a0218e3f6a14\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-szfml"
Oct 30 20:43:00.197619 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:43:00.197589    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjm8w\" (UniqueName: \"kubernetes.io/projected/9e9617c7-d6d3-447b-8dc2-a0218e3f6a14-kube-api-access-gjm8w\") pod \"kuadrant-operator-controller-manager-678d87766f-szfml\" (UID: \"9e9617c7-d6d3-447b-8dc2-a0218e3f6a14\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-szfml"
Oct 30 20:43:00.267383 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:43:00.267288    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-szfml"
Oct 30 20:43:00.573055 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:43:00.572975    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-szfml"]
Oct 30 20:43:00.576283 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:43:00.576251    4521 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e9617c7_d6d3_447b_8dc2_a0218e3f6a14.slice/crio-a268f2a47edb518a672cf31ea6acb3940b8e4353287846c73f0363c481a593a0 WatchSource:0}: Error finding container a268f2a47edb518a672cf31ea6acb3940b8e4353287846c73f0363c481a593a0: Status 404 returned error can't find the container with id a268f2a47edb518a672cf31ea6acb3940b8e4353287846c73f0363c481a593a0
Oct 30 20:43:01.205104 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:43:01.205069    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-szfml" event={"ID":"9e9617c7-d6d3-447b-8dc2-a0218e3f6a14","Type":"ContainerStarted","Data":"a268f2a47edb518a672cf31ea6acb3940b8e4353287846c73f0363c481a593a0"}
Oct 30 20:43:02.052138 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:43:02.052096    4521 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-fz9qs"]
Oct 30 20:43:02.055610 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:43:02.055584    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-fz9qs"
Oct 30 20:43:02.070499 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:43:02.070471    4521 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-rrt7j\""
Oct 30 20:43:02.073200 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:43:02.073178    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-fz9qs"]
Oct 30 20:43:02.096233 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:43:02.096201    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bl88\" (UniqueName: \"kubernetes.io/projected/91b6f058-7deb-4263-baae-9ed0ad3e7380-kube-api-access-7bl88\") pod \"limitador-operator-controller-manager-5778bb4449-fz9qs\" (UID: \"91b6f058-7deb-4263-baae-9ed0ad3e7380\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-fz9qs"
Oct 30 20:43:02.196735 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:43:02.196627    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bl88\" (UniqueName: \"kubernetes.io/projected/91b6f058-7deb-4263-baae-9ed0ad3e7380-kube-api-access-7bl88\") pod \"limitador-operator-controller-manager-5778bb4449-fz9qs\" (UID: \"91b6f058-7deb-4263-baae-9ed0ad3e7380\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-fz9qs"
Oct 30 20:43:02.210036 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:43:02.209998    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bl88\" (UniqueName: \"kubernetes.io/projected/91b6f058-7deb-4263-baae-9ed0ad3e7380-kube-api-access-7bl88\") pod \"limitador-operator-controller-manager-5778bb4449-fz9qs\" (UID: \"91b6f058-7deb-4263-baae-9ed0ad3e7380\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-fz9qs"
Oct 30 20:43:02.372386 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:43:02.372279    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-fz9qs"
Oct 30 20:43:02.672403 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:43:02.672260    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-fz9qs"]
Oct 30 20:43:02.674663 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:43:02.674637    4521 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91b6f058_7deb_4263_baae_9ed0ad3e7380.slice/crio-3e62fd0bc8c05204578f608afad341f9d45167ca4e3a5300348b6e0fb0f3a613 WatchSource:0}: Error finding container 3e62fd0bc8c05204578f608afad341f9d45167ca4e3a5300348b6e0fb0f3a613: Status 404 returned error can't find the container with id 3e62fd0bc8c05204578f608afad341f9d45167ca4e3a5300348b6e0fb0f3a613
Oct 30 20:43:03.217167 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:43:03.217127    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-fz9qs" event={"ID":"91b6f058-7deb-4263-baae-9ed0ad3e7380","Type":"ContainerStarted","Data":"3e62fd0bc8c05204578f608afad341f9d45167ca4e3a5300348b6e0fb0f3a613"}
Oct 30 20:43:05.229987 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:43:05.229952    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-szfml" event={"ID":"9e9617c7-d6d3-447b-8dc2-a0218e3f6a14","Type":"ContainerStarted","Data":"5b0aaf7421b560fb4ab09bdaa9edaff65d89d8531303f643aad221fd43d2ca93"}
Oct 30 20:43:05.230553 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:43:05.230129    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-szfml"
Oct 30 20:43:05.231755 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:43:05.231718    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-fz9qs" event={"ID":"91b6f058-7deb-4263-baae-9ed0ad3e7380","Type":"ContainerStarted","Data":"dd5e21d9e5c58583fd00663ae5f62f86374f10f6ade56da316345994c454a995"}
Oct 30 20:43:05.231855 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:43:05.231770    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-fz9qs"
Oct 30 20:43:05.257574 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:43:05.257522    4521 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-szfml" podStartSLOduration=2.342998326 podStartE2EDuration="6.257505654s" totalImagesPullingTime="3.914507328s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:42:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:43:05.254700503 +0000 UTC m=+252.600321739" watchObservedRunningTime="2026-10-30 20:43:05.257505654 +0000 UTC m=+252.603126891"
Oct 30 20:43:05.277295 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:43:05.277248    4521 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-fz9qs" podStartSLOduration=1.684690971 podStartE2EDuration="3.277234154s" totalImagesPullingTime="1.592543183s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:43:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:43:05.276126353 +0000 UTC m=+252.621747590" watchObservedRunningTime="2026-10-30 20:43:05.277234154 +0000 UTC m=+252.622855392"
Oct 30 20:43:06.578749 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:43:06.578710    4521 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-7jhkd"]
Oct 30 20:43:06.587500 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:43:06.587473    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-7jhkd"
Oct 30 20:43:06.593746 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:43:06.593722    4521 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-c9xgt\""
Oct 30 20:43:06.611597 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:43:06.611558    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-7jhkd"]
Oct 30 20:43:06.631308 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:43:06.631262    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8eba7406-6c06-4fa4-8022-0de75cfdbeb6-istio-token\") pod \"data-science-gateway-data-science-gateway-class-c574b7b6f-7jhkd\" (UID: \"8eba7406-6c06-4fa4-8022-0de75cfdbeb6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-7jhkd"
Oct 30 20:43:06.631516 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:43:06.631351    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8eba7406-6c06-4fa4-8022-0de75cfdbeb6-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-c574b7b6f-7jhkd\" (UID: \"8eba7406-6c06-4fa4-8022-0de75cfdbeb6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-7jhkd"
Oct 30 20:43:06.631516 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:43:06.631459    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8eba7406-6c06-4fa4-8022-0de75cfdbeb6-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-c574b7b6f-7jhkd\" (UID: \"8eba7406-6c06-4fa4-8022-0de75cfdbeb6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-7jhkd"
Oct 30 20:43:06.631597 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:43:06.631530    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8eba7406-6c06-4fa4-8022-0de75cfdbeb6-istio-data\") pod \"data-science-gateway-data-science-gateway-class-c574b7b6f-7jhkd\" (UID: \"8eba7406-6c06-4fa4-8022-0de75cfdbeb6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-7jhkd"
Oct 30 20:43:06.631597 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:43:06.631582    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8eba7406-6c06-4fa4-8022-0de75cfdbeb6-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-c574b7b6f-7jhkd\" (UID: \"8eba7406-6c06-4fa4-8022-0de75cfdbeb6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-7jhkd"
Oct 30 20:43:06.631684 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:43:06.631613    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmshm\" (UniqueName: \"kubernetes.io/projected/8eba7406-6c06-4fa4-8022-0de75cfdbeb6-kube-api-access-pmshm\") pod \"data-science-gateway-data-science-gateway-class-c574b7b6f-7jhkd\" (UID: \"8eba7406-6c06-4fa4-8022-0de75cfdbeb6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-7jhkd"
Oct 30 20:43:06.631684 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:43:06.631661    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8eba7406-6c06-4fa4-8022-0de75cfdbeb6-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-c574b7b6f-7jhkd\" (UID: \"8eba7406-6c06-4fa4-8022-0de75cfdbeb6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-7jhkd"
Oct 30 20:43:06.631761 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:43:06.631721    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8eba7406-6c06-4fa4-8022-0de75cfdbeb6-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-c574b7b6f-7jhkd\" (UID: \"8eba7406-6c06-4fa4-8022-0de75cfdbeb6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-7jhkd"
Oct 30 20:43:06.631761 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:43:06.631752    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8eba7406-6c06-4fa4-8022-0de75cfdbeb6-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-c574b7b6f-7jhkd\" (UID: \"8eba7406-6c06-4fa4-8022-0de75cfdbeb6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-7jhkd"
Oct 30 20:43:06.732506 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:43:06.732472    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8eba7406-6c06-4fa4-8022-0de75cfdbeb6-istio-token\") pod \"data-science-gateway-data-science-gateway-class-c574b7b6f-7jhkd\" (UID: \"8eba7406-6c06-4fa4-8022-0de75cfdbeb6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-7jhkd"
Oct 30 20:43:06.732506 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:43:06.732516    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8eba7406-6c06-4fa4-8022-0de75cfdbeb6-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-c574b7b6f-7jhkd\" (UID: \"8eba7406-6c06-4fa4-8022-0de75cfdbeb6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-7jhkd"
Oct 30 20:43:06.732858 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:43:06.732643    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8eba7406-6c06-4fa4-8022-0de75cfdbeb6-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-c574b7b6f-7jhkd\" (UID: \"8eba7406-6c06-4fa4-8022-0de75cfdbeb6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-7jhkd"
Oct 30 20:43:06.732858 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:43:06.732730    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8eba7406-6c06-4fa4-8022-0de75cfdbeb6-istio-data\") pod \"data-science-gateway-data-science-gateway-class-c574b7b6f-7jhkd\" (UID: \"8eba7406-6c06-4fa4-8022-0de75cfdbeb6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-7jhkd"
Oct 30 20:43:06.732858 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:43:06.732785    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8eba7406-6c06-4fa4-8022-0de75cfdbeb6-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-c574b7b6f-7jhkd\" (UID: \"8eba7406-6c06-4fa4-8022-0de75cfdbeb6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-7jhkd"
Oct 30 20:43:06.732858 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:43:06.732819    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmshm\" (UniqueName: \"kubernetes.io/projected/8eba7406-6c06-4fa4-8022-0de75cfdbeb6-kube-api-access-pmshm\") pod \"data-science-gateway-data-science-gateway-class-c574b7b6f-7jhkd\" (UID: \"8eba7406-6c06-4fa4-8022-0de75cfdbeb6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-7jhkd"
Oct 30 20:43:06.733275 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:43:06.732863    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8eba7406-6c06-4fa4-8022-0de75cfdbeb6-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-c574b7b6f-7jhkd\" (UID: \"8eba7406-6c06-4fa4-8022-0de75cfdbeb6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-7jhkd"
Oct 30 20:43:06.733275 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:43:06.732902    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8eba7406-6c06-4fa4-8022-0de75cfdbeb6-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-c574b7b6f-7jhkd\" (UID: \"8eba7406-6c06-4fa4-8022-0de75cfdbeb6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-7jhkd"
Oct 30 20:43:06.733275 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:43:06.732906    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8eba7406-6c06-4fa4-8022-0de75cfdbeb6-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-c574b7b6f-7jhkd\" (UID: \"8eba7406-6c06-4fa4-8022-0de75cfdbeb6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-7jhkd"
Oct 30 20:43:06.733275 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:43:06.732950    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8eba7406-6c06-4fa4-8022-0de75cfdbeb6-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-c574b7b6f-7jhkd\" (UID: \"8eba7406-6c06-4fa4-8022-0de75cfdbeb6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-7jhkd"
Oct 30 20:43:06.733275 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:43:06.733115    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8eba7406-6c06-4fa4-8022-0de75cfdbeb6-istio-data\") pod \"data-science-gateway-data-science-gateway-class-c574b7b6f-7jhkd\" (UID: \"8eba7406-6c06-4fa4-8022-0de75cfdbeb6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-7jhkd"
Oct 30 20:43:06.733275 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:43:06.733204    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8eba7406-6c06-4fa4-8022-0de75cfdbeb6-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-c574b7b6f-7jhkd\" (UID: \"8eba7406-6c06-4fa4-8022-0de75cfdbeb6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-7jhkd"
Oct 30 20:43:06.733589 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:43:06.733361    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8eba7406-6c06-4fa4-8022-0de75cfdbeb6-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-c574b7b6f-7jhkd\" (UID: \"8eba7406-6c06-4fa4-8022-0de75cfdbeb6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-7jhkd"
Oct 30 20:43:06.733800 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:43:06.733776    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8eba7406-6c06-4fa4-8022-0de75cfdbeb6-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-c574b7b6f-7jhkd\" (UID: \"8eba7406-6c06-4fa4-8022-0de75cfdbeb6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-7jhkd"
Oct 30 20:43:06.735436 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:43:06.735410    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8eba7406-6c06-4fa4-8022-0de75cfdbeb6-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-c574b7b6f-7jhkd\" (UID: \"8eba7406-6c06-4fa4-8022-0de75cfdbeb6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-7jhkd"
Oct 30 20:43:06.735626 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:43:06.735607    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8eba7406-6c06-4fa4-8022-0de75cfdbeb6-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-c574b7b6f-7jhkd\" (UID: \"8eba7406-6c06-4fa4-8022-0de75cfdbeb6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-7jhkd"
Oct 30 20:43:06.746240 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:43:06.746214    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmshm\" (UniqueName: \"kubernetes.io/projected/8eba7406-6c06-4fa4-8022-0de75cfdbeb6-kube-api-access-pmshm\") pod \"data-science-gateway-data-science-gateway-class-c574b7b6f-7jhkd\" (UID: \"8eba7406-6c06-4fa4-8022-0de75cfdbeb6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-7jhkd"
Oct 30 20:43:06.746465 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:43:06.746443    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8eba7406-6c06-4fa4-8022-0de75cfdbeb6-istio-token\") pod \"data-science-gateway-data-science-gateway-class-c574b7b6f-7jhkd\" (UID: \"8eba7406-6c06-4fa4-8022-0de75cfdbeb6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-7jhkd"
Oct 30 20:43:06.900193 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:43:06.900109    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-7jhkd"
Oct 30 20:43:07.187360 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:43:07.187306    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-x9bck"
Oct 30 20:43:07.236222 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:43:07.236188    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-7jhkd"]
Oct 30 20:43:07.239595 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:43:07.239567    4521 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8eba7406_6c06_4fa4_8022_0de75cfdbeb6.slice/crio-a3f733def982e224fb788bad639bd2e3d11d4fc0741c817082d7aa13938fea6a WatchSource:0}: Error finding container a3f733def982e224fb788bad639bd2e3d11d4fc0741c817082d7aa13938fea6a: Status 404 returned error can't find the container with id a3f733def982e224fb788bad639bd2e3d11d4fc0741c817082d7aa13938fea6a
Oct 30 20:43:07.242353 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:43:07.242285    4521 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:43:07.242459 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:43:07.242401    4521 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:43:07.242520 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:43:07.242475    4521 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:43:08.191849 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:43:08.191818    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-f97mf"
Oct 30 20:43:08.250635 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:43:08.250604    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-7jhkd" event={"ID":"8eba7406-6c06-4fa4-8022-0de75cfdbeb6","Type":"ContainerStarted","Data":"a029d9bc9bd7bcfc6e6b0cf14e17416bb51d28ea1cb6bf2f19d61598c91482c6"}
Oct 30 20:43:08.250804 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:43:08.250653    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-7jhkd" event={"ID":"8eba7406-6c06-4fa4-8022-0de75cfdbeb6","Type":"ContainerStarted","Data":"a3f733def982e224fb788bad639bd2e3d11d4fc0741c817082d7aa13938fea6a"}
Oct 30 20:43:08.350541 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:43:08.350488    4521 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-7jhkd" podStartSLOduration=2.350470579 podStartE2EDuration="2.350470579s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:43:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:43:08.348773509 +0000 UTC m=+255.694394760" watchObservedRunningTime="2026-10-30 20:43:08.350470579 +0000 UTC m=+255.696091873"
Oct 30 20:43:08.901050 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:43:08.901008    4521 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-7jhkd"
Oct 30 20:43:08.906081 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:43:08.906055    4521 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-7jhkd"
Oct 30 20:43:09.255859 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:43:09.255828    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-7jhkd"
Oct 30 20:43:09.256834 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:43:09.256815    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-7jhkd"
Oct 30 20:43:10.432817 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:43:10.432787    4521 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-vgh86"]
Oct 30 20:43:10.435691 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:43:10.435671    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-vgh86"
Oct 30 20:43:10.450414 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:43:10.450393    4521 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Oct 30 20:43:10.450877 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:43:10.450846    4521 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Oct 30 20:43:10.450877 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:43:10.450866    4521 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-g6g92\""
Oct 30 20:43:10.455790 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:43:10.455764    4521 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\""
Oct 30 20:43:10.457842 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:43:10.457821    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-vgh86"]
Oct 30 20:43:10.465095 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:43:10.465058    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f451091e-4af7-412a-8e0b-8cc42c1726ec-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-vgh86\" (UID: \"f451091e-4af7-412a-8e0b-8cc42c1726ec\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-vgh86"
Oct 30 20:43:10.465223 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:43:10.465096    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4b26l\" (UniqueName: \"kubernetes.io/projected/f451091e-4af7-412a-8e0b-8cc42c1726ec-kube-api-access-4b26l\") pod \"kserve-module-controller-manager-7566cf5fc8-vgh86\" (UID: \"f451091e-4af7-412a-8e0b-8cc42c1726ec\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-vgh86"
Oct 30 20:43:10.465317 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:43:10.465289    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f451091e-4af7-412a-8e0b-8cc42c1726ec-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-vgh86\" (UID: \"f451091e-4af7-412a-8e0b-8cc42c1726ec\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-vgh86"
Oct 30 20:43:10.465418 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:43:10.465397    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f451091e-4af7-412a-8e0b-8cc42c1726ec-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-vgh86\" (UID: \"f451091e-4af7-412a-8e0b-8cc42c1726ec\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-vgh86"
Oct 30 20:43:10.566452 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:43:10.566415    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f451091e-4af7-412a-8e0b-8cc42c1726ec-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-vgh86\" (UID: \"f451091e-4af7-412a-8e0b-8cc42c1726ec\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-vgh86"
Oct 30 20:43:10.566452 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:43:10.566456    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4b26l\" (UniqueName: \"kubernetes.io/projected/f451091e-4af7-412a-8e0b-8cc42c1726ec-kube-api-access-4b26l\") pod \"kserve-module-controller-manager-7566cf5fc8-vgh86\" (UID: \"f451091e-4af7-412a-8e0b-8cc42c1726ec\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-vgh86"
Oct 30 20:43:10.566636 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:43:10.566512    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f451091e-4af7-412a-8e0b-8cc42c1726ec-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-vgh86\" (UID: \"f451091e-4af7-412a-8e0b-8cc42c1726ec\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-vgh86"
Oct 30 20:43:10.566636 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:43:10.566536    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f451091e-4af7-412a-8e0b-8cc42c1726ec-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-vgh86\" (UID: \"f451091e-4af7-412a-8e0b-8cc42c1726ec\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-vgh86"
Oct 30 20:43:10.566918 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:43:10.566896    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f451091e-4af7-412a-8e0b-8cc42c1726ec-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-vgh86\" (UID: \"f451091e-4af7-412a-8e0b-8cc42c1726ec\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-vgh86"
Oct 30 20:43:10.566951 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:43:10.566922    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f451091e-4af7-412a-8e0b-8cc42c1726ec-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-vgh86\" (UID: \"f451091e-4af7-412a-8e0b-8cc42c1726ec\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-vgh86"
Oct 30 20:43:10.567401 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:43:10.567383    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f451091e-4af7-412a-8e0b-8cc42c1726ec-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-vgh86\" (UID: \"f451091e-4af7-412a-8e0b-8cc42c1726ec\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-vgh86"
Oct 30 20:43:10.591509 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:43:10.591475    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4b26l\" (UniqueName: \"kubernetes.io/projected/f451091e-4af7-412a-8e0b-8cc42c1726ec-kube-api-access-4b26l\") pod \"kserve-module-controller-manager-7566cf5fc8-vgh86\" (UID: \"f451091e-4af7-412a-8e0b-8cc42c1726ec\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-vgh86"
Oct 30 20:43:10.747718 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:43:10.747678    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-vgh86"
Oct 30 20:43:11.026623 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:43:11.026596    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-vgh86"]
Oct 30 20:43:11.028419 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:43:11.028392    4521 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf451091e_4af7_412a_8e0b_8cc42c1726ec.slice/crio-ab3cbdc452650b10bff93540bb71cd8072bd007610202acb75d981e84a0a852f WatchSource:0}: Error finding container ab3cbdc452650b10bff93540bb71cd8072bd007610202acb75d981e84a0a852f: Status 404 returned error can't find the container with id ab3cbdc452650b10bff93540bb71cd8072bd007610202acb75d981e84a0a852f
Oct 30 20:43:11.267740 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:43:11.267707    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-vgh86" event={"ID":"f451091e-4af7-412a-8e0b-8cc42c1726ec","Type":"ContainerStarted","Data":"ab3cbdc452650b10bff93540bb71cd8072bd007610202acb75d981e84a0a852f"}
Oct 30 20:43:14.282901 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:43:14.282858    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-vgh86" event={"ID":"f451091e-4af7-412a-8e0b-8cc42c1726ec","Type":"ContainerStarted","Data":"1063461c17a50c1efb75187d2af797c0de1d5f6e9faa32392e9ae38cebf469d4"}
Oct 30 20:43:14.283429 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:43:14.282986    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-vgh86"
Oct 30 20:43:14.309513 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:43:14.309458    4521 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-vgh86" podStartSLOduration=2.020194796 podStartE2EDuration="4.309445266s" totalImagesPullingTime="2.28925047s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:43:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:43:14.309074605 +0000 UTC m=+261.654695844" watchObservedRunningTime="2026-10-30 20:43:14.309445266 +0000 UTC m=+261.655066505"
Oct 30 20:43:16.240385 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:43:16.240356    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-fz9qs"
Oct 30 20:43:16.240835 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:43:16.240441    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-szfml"
Oct 30 20:43:16.429042 ip-10-0-131-125 kubenswrapper[4521]: E1030 20:43:16.429003    4521 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lkn2k" containerName="extractor"
Oct 30 20:43:35.289661 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:43:35.289629    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-vgh86"
Oct 30 20:43:46.430348 ip-10-0-131-125 kubenswrapper[4521]: E1030 20:43:46.430298    4521 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lkn2k" containerName="extractor"
Oct 30 20:44:04.412421 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:44:04.412378    4521 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-kmh84"]
Oct 30 20:44:04.423063 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:44:04.423031    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-kmh84"
Oct 30 20:44:04.431348 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:44:04.431307    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-kmh84"]
Oct 30 20:44:04.442865 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:44:04.442835    4521 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-9fpcz\""
Oct 30 20:44:04.443147 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:44:04.443128    4521 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Oct 30 20:44:04.544625 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:44:04.544594    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/eb9ee8df-9133-41fe-8128-cb1c28486d6b-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-kmh84\" (UID: \"eb9ee8df-9133-41fe-8128-cb1c28486d6b\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-kmh84"
Oct 30 20:44:04.544800 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:44:04.544630    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4p5x\" (UniqueName: \"kubernetes.io/projected/eb9ee8df-9133-41fe-8128-cb1c28486d6b-kube-api-access-w4p5x\") pod \"llmisvc-controller-manager-869d5dcc5d-kmh84\" (UID: \"eb9ee8df-9133-41fe-8128-cb1c28486d6b\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-kmh84"
Oct 30 20:44:04.645417 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:44:04.645313    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/eb9ee8df-9133-41fe-8128-cb1c28486d6b-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-kmh84\" (UID: \"eb9ee8df-9133-41fe-8128-cb1c28486d6b\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-kmh84"
Oct 30 20:44:04.645417 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:44:04.645396    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4p5x\" (UniqueName: \"kubernetes.io/projected/eb9ee8df-9133-41fe-8128-cb1c28486d6b-kube-api-access-w4p5x\") pod \"llmisvc-controller-manager-869d5dcc5d-kmh84\" (UID: \"eb9ee8df-9133-41fe-8128-cb1c28486d6b\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-kmh84"
Oct 30 20:44:04.647747 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:44:04.647721    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/eb9ee8df-9133-41fe-8128-cb1c28486d6b-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-kmh84\" (UID: \"eb9ee8df-9133-41fe-8128-cb1c28486d6b\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-kmh84"
Oct 30 20:44:04.661369 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:44:04.661344    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4p5x\" (UniqueName: \"kubernetes.io/projected/eb9ee8df-9133-41fe-8128-cb1c28486d6b-kube-api-access-w4p5x\") pod \"llmisvc-controller-manager-869d5dcc5d-kmh84\" (UID: \"eb9ee8df-9133-41fe-8128-cb1c28486d6b\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-kmh84"
Oct 30 20:44:04.739518 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:44:04.739482    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-kmh84"
Oct 30 20:44:05.026741 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:44:05.026710    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-kmh84"]
Oct 30 20:44:05.028637 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:44:05.028607    4521 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podeb9ee8df_9133_41fe_8128_cb1c28486d6b.slice/crio-6f370fc5a08f4e9ee729544f0db6a147d88fc8912ecf52f1a5c22079bf00ba70 WatchSource:0}: Error finding container 6f370fc5a08f4e9ee729544f0db6a147d88fc8912ecf52f1a5c22079bf00ba70: Status 404 returned error can't find the container with id 6f370fc5a08f4e9ee729544f0db6a147d88fc8912ecf52f1a5c22079bf00ba70
Oct 30 20:44:05.030433 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:44:05.030415    4521 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 20:44:05.534205 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:44:05.534166    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-kmh84" event={"ID":"eb9ee8df-9133-41fe-8128-cb1c28486d6b","Type":"ContainerStarted","Data":"6f370fc5a08f4e9ee729544f0db6a147d88fc8912ecf52f1a5c22079bf00ba70"}
Oct 30 20:44:07.551529 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:44:07.551492    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-kmh84" event={"ID":"eb9ee8df-9133-41fe-8128-cb1c28486d6b","Type":"ContainerStarted","Data":"6daaed1f41d3b0a7db4af093b8795ea1c92f2461889f85e2c42f2a82c3710259"}
Oct 30 20:44:07.552021 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:44:07.551691    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-kmh84"
Oct 30 20:44:07.571956 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:44:07.571908    4521 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-kmh84" podStartSLOduration=1.9104863619999999 podStartE2EDuration="3.571894842s" totalImagesPullingTime="1.66140848s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:44:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:44:07.571543012 +0000 UTC m=+314.917164250" watchObservedRunningTime="2026-10-30 20:44:07.571894842 +0000 UTC m=+314.917516080"
Oct 30 20:44:16.428120 ip-10-0-131-125 kubenswrapper[4521]: E1030 20:44:16.428079    4521 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lkn2k" containerName="extractor"
Oct 30 20:44:38.559476 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:44:38.559441    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-kmh84"
Oct 30 20:44:46.429573 ip-10-0-131-125 kubenswrapper[4521]: E1030 20:44:46.429535    4521 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lkn2k" containerName="extractor"
Oct 30 20:45:16.428347 ip-10-0-131-125 kubenswrapper[4521]: E1030 20:45:16.428292    4521 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lkn2k" containerName="extractor"
Oct 30 20:45:46.428263 ip-10-0-131-125 kubenswrapper[4521]: E1030 20:45:46.428223    4521 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lkn2k" containerName="extractor"
Oct 30 20:46:16.428484 ip-10-0-131-125 kubenswrapper[4521]: E1030 20:46:16.428447    4521 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lkn2k" containerName="extractor"
Oct 30 20:46:46.428611 ip-10-0-131-125 kubenswrapper[4521]: E1030 20:46:46.428573    4521 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lkn2k" containerName="extractor"
Oct 30 20:47:16.430564 ip-10-0-131-125 kubenswrapper[4521]: E1030 20:47:16.430477    4521 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lkn2k" containerName="extractor"
Oct 30 20:47:46.428196 ip-10-0-131-125 kubenswrapper[4521]: E1030 20:47:46.428145    4521 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lkn2k" containerName="extractor"
Oct 30 20:48:16.431688 ip-10-0-131-125 kubenswrapper[4521]: E1030 20:48:16.431643    4521 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lkn2k" containerName="extractor"
Oct 30 20:48:46.427722 ip-10-0-131-125 kubenswrapper[4521]: E1030 20:48:46.427686    4521 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lkn2k" containerName="extractor"
Oct 30 20:49:16.428514 ip-10-0-131-125 kubenswrapper[4521]: E1030 20:49:16.428474    4521 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lkn2k" containerName="extractor"
Oct 30 20:49:46.428469 ip-10-0-131-125 kubenswrapper[4521]: E1030 20:49:46.428435    4521 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lkn2k" containerName="extractor"
Oct 30 20:50:16.428691 ip-10-0-131-125 kubenswrapper[4521]: E1030 20:50:16.428608    4521 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lkn2k" containerName="extractor"
Oct 30 20:50:46.431424 ip-10-0-131-125 kubenswrapper[4521]: E1030 20:50:46.431385    4521 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lkn2k" containerName="extractor"
Oct 30 20:51:16.428435 ip-10-0-131-125 kubenswrapper[4521]: E1030 20:51:16.428396    4521 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lkn2k" containerName="extractor"
Oct 30 20:51:46.429025 ip-10-0-131-125 kubenswrapper[4521]: E1030 20:51:46.428989    4521 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lkn2k" containerName="extractor"
Oct 30 20:52:16.428813 ip-10-0-131-125 kubenswrapper[4521]: E1030 20:52:16.428773    4521 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lkn2k" containerName="extractor"
Oct 30 20:52:46.428961 ip-10-0-131-125 kubenswrapper[4521]: E1030 20:52:46.428922    4521 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lkn2k" containerName="extractor"
Oct 30 20:53:16.429853 ip-10-0-131-125 kubenswrapper[4521]: E1030 20:53:16.429814    4521 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lkn2k" containerName="extractor"
Oct 30 20:53:26.924766 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:53:26.924719    4521 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-szfml_9e9617c7-d6d3-447b-8dc2-a0218e3f6a14/manager/0.log"
Oct 30 20:53:27.128850 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:53:27.128815    4521 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-szfml"]
Oct 30 20:53:27.129108 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:53:27.129064    4521 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-szfml" podUID="9e9617c7-d6d3-447b-8dc2-a0218e3f6a14" containerName="manager" containerID="cri-o://5b0aaf7421b560fb4ab09bdaa9edaff65d89d8531303f643aad221fd43d2ca93" gracePeriod=10
Oct 30 20:53:27.165538 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:53:27.165508    4521 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-qtt5r"]
Oct 30 20:53:27.168772 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:53:27.168753    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-qtt5r"
Oct 30 20:53:27.184237 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:53:27.184175    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-qtt5r"]
Oct 30 20:53:27.286117 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:53:27.286078    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rclh\" (UniqueName: \"kubernetes.io/projected/30233858-576c-450d-a5bf-f0d5ca1dd905-kube-api-access-2rclh\") pod \"kuadrant-operator-controller-manager-678d87766f-qtt5r\" (UID: \"30233858-576c-450d-a5bf-f0d5ca1dd905\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-qtt5r"
Oct 30 20:53:27.286310 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:53:27.286213    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/30233858-576c-450d-a5bf-f0d5ca1dd905-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-qtt5r\" (UID: \"30233858-576c-450d-a5bf-f0d5ca1dd905\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-qtt5r"
Oct 30 20:53:27.371537 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:53:27.371511    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-szfml"
Oct 30 20:53:27.387453 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:53:27.387420    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/30233858-576c-450d-a5bf-f0d5ca1dd905-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-qtt5r\" (UID: \"30233858-576c-450d-a5bf-f0d5ca1dd905\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-qtt5r"
Oct 30 20:53:27.387604 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:53:27.387482    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rclh\" (UniqueName: \"kubernetes.io/projected/30233858-576c-450d-a5bf-f0d5ca1dd905-kube-api-access-2rclh\") pod \"kuadrant-operator-controller-manager-678d87766f-qtt5r\" (UID: \"30233858-576c-450d-a5bf-f0d5ca1dd905\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-qtt5r"
Oct 30 20:53:27.387859 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:53:27.387838    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/30233858-576c-450d-a5bf-f0d5ca1dd905-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-qtt5r\" (UID: \"30233858-576c-450d-a5bf-f0d5ca1dd905\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-qtt5r"
Oct 30 20:53:27.400116 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:53:27.400086    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rclh\" (UniqueName: \"kubernetes.io/projected/30233858-576c-450d-a5bf-f0d5ca1dd905-kube-api-access-2rclh\") pod \"kuadrant-operator-controller-manager-678d87766f-qtt5r\" (UID: \"30233858-576c-450d-a5bf-f0d5ca1dd905\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-qtt5r"
Oct 30 20:53:27.488472 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:53:27.488428    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e9617c7-d6d3-447b-8dc2-a0218e3f6a14-kube-api-access-gjm8w\" (UniqueName: \"kubernetes.io/projected/9e9617c7-d6d3-447b-8dc2-a0218e3f6a14-kube-api-access-gjm8w\") pod \"9e9617c7-d6d3-447b-8dc2-a0218e3f6a14\" (UID: \"9e9617c7-d6d3-447b-8dc2-a0218e3f6a14\") "
Oct 30 20:53:27.488680 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:53:27.488532    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e9617c7-d6d3-447b-8dc2-a0218e3f6a14-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9e9617c7-d6d3-447b-8dc2-a0218e3f6a14-extensions-socket-volume\") pod \"9e9617c7-d6d3-447b-8dc2-a0218e3f6a14\" (UID: \"9e9617c7-d6d3-447b-8dc2-a0218e3f6a14\") "
Oct 30 20:53:27.488943 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:53:27.488917    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e9617c7-d6d3-447b-8dc2-a0218e3f6a14-extensions-socket-volume" pod "9e9617c7-d6d3-447b-8dc2-a0218e3f6a14" (UID: "9e9617c7-d6d3-447b-8dc2-a0218e3f6a14"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:53:27.490612 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:53:27.490593    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e9617c7-d6d3-447b-8dc2-a0218e3f6a14-kube-api-access-gjm8w" pod "9e9617c7-d6d3-447b-8dc2-a0218e3f6a14" (UID: "9e9617c7-d6d3-447b-8dc2-a0218e3f6a14"). InnerVolumeSpecName "kube-api-access-gjm8w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:53:27.513856 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:53:27.513813    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-qtt5r"
Oct 30 20:53:27.589881 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:53:27.589847    4521 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjm8w\" (UniqueName: \"kubernetes.io/projected/9e9617c7-d6d3-447b-8dc2-a0218e3f6a14-kube-api-access-gjm8w\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:53:27.589881 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:53:27.589874    4521 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9e9617c7-d6d3-447b-8dc2-a0218e3f6a14-extensions-socket-volume\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:53:27.799922 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:53:27.799866    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-qtt5r"]
Oct 30 20:53:27.802234 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:53:27.802207    4521 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30233858_576c_450d_a5bf_f0d5ca1dd905.slice/crio-d4ee57970d9e132201a39bf2827575b3f832c3527a77c64c3cb64d0b725b1f18 WatchSource:0}: Error finding container d4ee57970d9e132201a39bf2827575b3f832c3527a77c64c3cb64d0b725b1f18: Status 404 returned error can't find the container with id d4ee57970d9e132201a39bf2827575b3f832c3527a77c64c3cb64d0b725b1f18
Oct 30 20:53:28.207007 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:53:28.206973    4521 generic.go:356] "Generic (PLEG): container finished" podID="9e9617c7-d6d3-447b-8dc2-a0218e3f6a14" containerID="5b0aaf7421b560fb4ab09bdaa9edaff65d89d8531303f643aad221fd43d2ca93" exitCode=0
Oct 30 20:53:28.207510 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:53:28.207019    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-szfml"
Oct 30 20:53:28.207510 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:53:28.207060    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-szfml" event={"ID":"9e9617c7-d6d3-447b-8dc2-a0218e3f6a14","Type":"ContainerDied","Data":"5b0aaf7421b560fb4ab09bdaa9edaff65d89d8531303f643aad221fd43d2ca93"}
Oct 30 20:53:28.207510 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:53:28.207112    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-szfml" event={"ID":"9e9617c7-d6d3-447b-8dc2-a0218e3f6a14","Type":"ContainerDied","Data":"a268f2a47edb518a672cf31ea6acb3940b8e4353287846c73f0363c481a593a0"}
Oct 30 20:53:28.207510 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:53:28.207135    4521 scope.go:122] "RemoveContainer" containerID="5b0aaf7421b560fb4ab09bdaa9edaff65d89d8531303f643aad221fd43d2ca93"
Oct 30 20:53:28.208860 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:53:28.208839    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-qtt5r" event={"ID":"30233858-576c-450d-a5bf-f0d5ca1dd905","Type":"ContainerStarted","Data":"f361ed9ec3e0dac475abc6eb24d31fda6f2e3f65e352696eed306ec28fe47f93"}
Oct 30 20:53:28.208948 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:53:28.208876    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-qtt5r" event={"ID":"30233858-576c-450d-a5bf-f0d5ca1dd905","Type":"ContainerStarted","Data":"d4ee57970d9e132201a39bf2827575b3f832c3527a77c64c3cb64d0b725b1f18"}
Oct 30 20:53:28.208948 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:53:28.208903    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-qtt5r"
Oct 30 20:53:28.217171 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:53:28.217154    4521 scope.go:122] "RemoveContainer" containerID="5b0aaf7421b560fb4ab09bdaa9edaff65d89d8531303f643aad221fd43d2ca93"
Oct 30 20:53:28.217475 ip-10-0-131-125 kubenswrapper[4521]: E1030 20:53:28.217452    4521 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b0aaf7421b560fb4ab09bdaa9edaff65d89d8531303f643aad221fd43d2ca93\": container with ID starting with 5b0aaf7421b560fb4ab09bdaa9edaff65d89d8531303f643aad221fd43d2ca93 not found: ID does not exist" containerID="5b0aaf7421b560fb4ab09bdaa9edaff65d89d8531303f643aad221fd43d2ca93"
Oct 30 20:53:28.217572 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:53:28.217481    4521 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b0aaf7421b560fb4ab09bdaa9edaff65d89d8531303f643aad221fd43d2ca93"} err="failed to get container status \"5b0aaf7421b560fb4ab09bdaa9edaff65d89d8531303f643aad221fd43d2ca93\": rpc error: code = NotFound desc = could not find container \"5b0aaf7421b560fb4ab09bdaa9edaff65d89d8531303f643aad221fd43d2ca93\": container with ID starting with 5b0aaf7421b560fb4ab09bdaa9edaff65d89d8531303f643aad221fd43d2ca93 not found: ID does not exist"
Oct 30 20:53:28.240795 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:53:28.240742    4521 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-qtt5r" podStartSLOduration=1.24072869 podStartE2EDuration="1.24072869s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:53:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:53:28.238051052 +0000 UTC m=+875.583672291" watchObservedRunningTime="2026-10-30 20:53:28.24072869 +0000 UTC m=+875.586349929"
Oct 30 20:53:28.256152 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:53:28.256120    4521 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-szfml"]
Oct 30 20:53:28.264378 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:53:28.264350    4521 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-szfml"]
Oct 30 20:53:28.790538 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:53:28.790506    4521 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e9617c7-d6d3-447b-8dc2-a0218e3f6a14" path="/var/lib/kubelet/pods/9e9617c7-d6d3-447b-8dc2-a0218e3f6a14/volumes"
Oct 30 20:53:39.216983 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:53:39.216951    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-qtt5r"
Oct 30 20:53:44.327857 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:53:44.327821    4521 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4kxb"]
Oct 30 20:53:44.328249 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:53:44.328153    4521 cpu_manager.go:414] "containerMap: removing container" podUID="9e9617c7-d6d3-447b-8dc2-a0218e3f6a14" containerName="manager"
Oct 30 20:53:44.328249 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:53:44.328164    4521 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e9617c7-d6d3-447b-8dc2-a0218e3f6a14" containerName="manager"
Oct 30 20:53:44.328249 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:53:44.328239    4521 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e9617c7-d6d3-447b-8dc2-a0218e3f6a14" containerName="manager"
Oct 30 20:53:44.331148 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:53:44.331132    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-f4kxb"
Oct 30 20:53:44.353793 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:53:44.353763    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4kxb"]
Oct 30 20:53:44.354319 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:53:44.354298    4521 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Oct 30 20:53:44.354319 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:53:44.354306    4521 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-qj27b\""
Oct 30 20:53:44.400999 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:53:44.400964    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4kxb"]
Oct 30 20:53:44.440104 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:53:44.440069    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdfrb\" (UniqueName: \"kubernetes.io/projected/3d5dbf9c-8743-4193-8c60-3f49a346776d-kube-api-access-zdfrb\") pod \"limitador-limitador-7c9b87d574-f4kxb\" (UID: \"3d5dbf9c-8743-4193-8c60-3f49a346776d\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-f4kxb"
Oct 30 20:53:44.440104 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:53:44.440103    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3d5dbf9c-8743-4193-8c60-3f49a346776d-config-file\") pod \"limitador-limitador-7c9b87d574-f4kxb\" (UID: \"3d5dbf9c-8743-4193-8c60-3f49a346776d\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-f4kxb"
Oct 30 20:53:44.541400 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:53:44.541360    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdfrb\" (UniqueName: \"kubernetes.io/projected/3d5dbf9c-8743-4193-8c60-3f49a346776d-kube-api-access-zdfrb\") pod \"limitador-limitador-7c9b87d574-f4kxb\" (UID: \"3d5dbf9c-8743-4193-8c60-3f49a346776d\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-f4kxb"
Oct 30 20:53:44.541400 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:53:44.541395    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3d5dbf9c-8743-4193-8c60-3f49a346776d-config-file\") pod \"limitador-limitador-7c9b87d574-f4kxb\" (UID: \"3d5dbf9c-8743-4193-8c60-3f49a346776d\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-f4kxb"
Oct 30 20:53:44.542265 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:53:44.542232    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3d5dbf9c-8743-4193-8c60-3f49a346776d-config-file\") pod \"limitador-limitador-7c9b87d574-f4kxb\" (UID: \"3d5dbf9c-8743-4193-8c60-3f49a346776d\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-f4kxb"
Oct 30 20:53:44.553868 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:53:44.553838    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdfrb\" (UniqueName: \"kubernetes.io/projected/3d5dbf9c-8743-4193-8c60-3f49a346776d-kube-api-access-zdfrb\") pod \"limitador-limitador-7c9b87d574-f4kxb\" (UID: \"3d5dbf9c-8743-4193-8c60-3f49a346776d\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-f4kxb"
Oct 30 20:53:44.643822 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:53:44.643744    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-f4kxb"
Oct 30 20:53:44.927935 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:53:44.927905    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4kxb"]
Oct 30 20:53:44.930189 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:53:44.930158    4521 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d5dbf9c_8743_4193_8c60_3f49a346776d.slice/crio-f92efb7a008aa8faa7abb0946afce7c668af51da749a288378382b0058b35089 WatchSource:0}: Error finding container f92efb7a008aa8faa7abb0946afce7c668af51da749a288378382b0058b35089: Status 404 returned error can't find the container with id f92efb7a008aa8faa7abb0946afce7c668af51da749a288378382b0058b35089
Oct 30 20:53:44.932461 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:53:44.932442    4521 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 20:53:45.296136 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:53:45.296102    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-f4kxb" event={"ID":"3d5dbf9c-8743-4193-8c60-3f49a346776d","Type":"ContainerStarted","Data":"f92efb7a008aa8faa7abb0946afce7c668af51da749a288378382b0058b35089"}
Oct 30 20:53:46.430503 ip-10-0-131-125 kubenswrapper[4521]: E1030 20:53:46.430466    4521 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lkn2k" containerName="extractor"
Oct 30 20:53:47.307375 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:53:47.307310    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-f4kxb" event={"ID":"3d5dbf9c-8743-4193-8c60-3f49a346776d","Type":"ContainerStarted","Data":"cce7d0748299e0657f36b0718e3c4dd9c823b7c4709eac97f4a612a23a68ebe1"}
Oct 30 20:53:47.307572 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:53:47.307417    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-f4kxb"
Oct 30 20:53:47.330031 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:53:47.329976    4521 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-f4kxb" podStartSLOduration=1.728599794 podStartE2EDuration="3.329962088s" totalImagesPullingTime="1.601362294s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:53:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:53:47.329571839 +0000 UTC m=+894.675193086" watchObservedRunningTime="2026-10-30 20:53:47.329962088 +0000 UTC m=+894.675583398"
Oct 30 20:53:58.313281 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:53:58.313251    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-f4kxb"
Oct 30 20:54:16.428864 ip-10-0-131-125 kubenswrapper[4521]: E1030 20:54:16.428827    4521 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lkn2k" containerName="extractor"
Oct 30 20:54:46.428541 ip-10-0-131-125 kubenswrapper[4521]: E1030 20:54:46.428492    4521 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lkn2k" containerName="extractor"
Oct 30 20:54:57.519017 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:54:57.518980    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4kxb"]
Oct 30 20:55:07.704536 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:55:07.704498    4521 generic.go:356] "Generic (PLEG): container finished" podID="62149acf-8605-4f8d-b1ac-6e8d6b4865fb" containerID="cba5b42bbb7946b983df7fbc9e84bee3ae8bd1ac22284f6c94104007a8604640" exitCode=0
Oct 30 20:55:07.705137 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:55:07.704572    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-bckjb" event={"ID":"62149acf-8605-4f8d-b1ac-6e8d6b4865fb","Type":"ContainerDied","Data":"cba5b42bbb7946b983df7fbc9e84bee3ae8bd1ac22284f6c94104007a8604640"}
Oct 30 20:55:07.705137 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:55:07.705028    4521 scope.go:122] "RemoveContainer" containerID="cba5b42bbb7946b983df7fbc9e84bee3ae8bd1ac22284f6c94104007a8604640"
Oct 30 20:55:07.706107 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:55:07.706080    4521 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:55:07.706190 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:55:07.706145    4521 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:55:07.706190 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:55:07.706177    4521 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:55:07.732671 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:55:07.732646    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-bckjb"
Oct 30 20:55:08.736687 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:55:08.736644    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-bckjb" event={"ID":"62149acf-8605-4f8d-b1ac-6e8d6b4865fb","Type":"ContainerStarted","Data":"ec018059ccfdf6a7613d15e8bc04c30c765bc3f12ac4b1ac77fc32f4680639cf"}
Oct 30 20:55:08.940175 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:55:08.940139    4521 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-bckjb"
Oct 30 20:55:08.941205 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:55:08.941175    4521 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-bckjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused" start-of-body=
Oct 30 20:55:08.941316 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:55:08.941229    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-bckjb" podUID="62149acf-8605-4f8d-b1ac-6e8d6b4865fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused"
Oct 30 20:55:09.940880 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:55:09.940841    4521 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-bckjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused" start-of-body=
Oct 30 20:55:09.941314 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:55:09.940903    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-bckjb" podUID="62149acf-8605-4f8d-b1ac-6e8d6b4865fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused"
Oct 30 20:55:10.271050 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:55:10.270963    4521 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-vtzgz"]
Oct 30 20:55:10.273515 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:55:10.273491    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-vtzgz"
Oct 30 20:55:10.286016 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:55:10.285989    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-vtzgz"]
Oct 30 20:55:10.388413 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:55:10.388366    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9spbw\" (UniqueName: \"kubernetes.io/projected/d306b705-56e3-4de0-a4cb-fe23b9621cf0-kube-api-access-9spbw\") pod \"llmisvc-controller-manager-6466c554d8-vtzgz\" (UID: \"d306b705-56e3-4de0-a4cb-fe23b9621cf0\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-vtzgz"
Oct 30 20:55:10.388607 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:55:10.388498    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d306b705-56e3-4de0-a4cb-fe23b9621cf0-cert\") pod \"llmisvc-controller-manager-6466c554d8-vtzgz\" (UID: \"d306b705-56e3-4de0-a4cb-fe23b9621cf0\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-vtzgz"
Oct 30 20:55:10.489649 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:55:10.489608    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d306b705-56e3-4de0-a4cb-fe23b9621cf0-cert\") pod \"llmisvc-controller-manager-6466c554d8-vtzgz\" (UID: \"d306b705-56e3-4de0-a4cb-fe23b9621cf0\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-vtzgz"
Oct 30 20:55:10.489864 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:55:10.489692    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9spbw\" (UniqueName: \"kubernetes.io/projected/d306b705-56e3-4de0-a4cb-fe23b9621cf0-kube-api-access-9spbw\") pod \"llmisvc-controller-manager-6466c554d8-vtzgz\" (UID: \"d306b705-56e3-4de0-a4cb-fe23b9621cf0\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-vtzgz"
Oct 30 20:55:10.492195 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:55:10.492168    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d306b705-56e3-4de0-a4cb-fe23b9621cf0-cert\") pod \"llmisvc-controller-manager-6466c554d8-vtzgz\" (UID: \"d306b705-56e3-4de0-a4cb-fe23b9621cf0\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-vtzgz"
Oct 30 20:55:10.502620 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:55:10.502587    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9spbw\" (UniqueName: \"kubernetes.io/projected/d306b705-56e3-4de0-a4cb-fe23b9621cf0-kube-api-access-9spbw\") pod \"llmisvc-controller-manager-6466c554d8-vtzgz\" (UID: \"d306b705-56e3-4de0-a4cb-fe23b9621cf0\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-vtzgz"
Oct 30 20:55:10.585516 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:55:10.585431    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-vtzgz"
Oct 30 20:55:10.880548 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:55:10.880518    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-vtzgz"]
Oct 30 20:55:10.882016 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:55:10.881990    4521 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd306b705_56e3_4de0_a4cb_fe23b9621cf0.slice/crio-3cb80d71dda1aeda702d3f37d542307f166575540b5cd8c5fda0454e2b55a579 WatchSource:0}: Error finding container 3cb80d71dda1aeda702d3f37d542307f166575540b5cd8c5fda0454e2b55a579: Status 404 returned error can't find the container with id 3cb80d71dda1aeda702d3f37d542307f166575540b5cd8c5fda0454e2b55a579
Oct 30 20:55:10.940878 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:55:10.940823    4521 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-bckjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused" start-of-body=
Oct 30 20:55:10.941076 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:55:10.940900    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-bckjb" podUID="62149acf-8605-4f8d-b1ac-6e8d6b4865fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused"
Oct 30 20:55:11.753953 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:55:11.753915    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-vtzgz" event={"ID":"d306b705-56e3-4de0-a4cb-fe23b9621cf0","Type":"ContainerStarted","Data":"9758a5845b6159b5313ba1a6d26b62fe324ff9a26f92a14fe73f2db5f475d8c7"}
Oct 30 20:55:11.754143 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:55:11.753975    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-vtzgz" event={"ID":"d306b705-56e3-4de0-a4cb-fe23b9621cf0","Type":"ContainerStarted","Data":"3cb80d71dda1aeda702d3f37d542307f166575540b5cd8c5fda0454e2b55a579"}
Oct 30 20:55:11.754143 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:55:11.754001    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-vtzgz"
Oct 30 20:55:11.774388 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:55:11.774334    4521 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-vtzgz" podStartSLOduration=1.365027686 podStartE2EDuration="1.77430431s" totalImagesPullingTime="409.276624ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:55:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:55:11.773423144 +0000 UTC m=+979.119044375" watchObservedRunningTime="2026-10-30 20:55:11.77430431 +0000 UTC m=+979.119925547"
Oct 30 20:55:11.940763 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:55:11.940721    4521 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-bckjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused" start-of-body=
Oct 30 20:55:11.940920 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:55:11.940783    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-bckjb" podUID="62149acf-8605-4f8d-b1ac-6e8d6b4865fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused"
Oct 30 20:55:12.940666 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:55:12.940629    4521 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-bckjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused" start-of-body=
Oct 30 20:55:12.941056 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:55:12.940688    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-bckjb" podUID="62149acf-8605-4f8d-b1ac-6e8d6b4865fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused"
Oct 30 20:55:13.944227 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:55:13.944197    4521 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-bckjb"
Oct 30 20:55:13.944634 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:55:13.944524    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-bckjb"
Oct 30 20:55:13.945183 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:55:13.945168    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-bckjb"
Oct 30 20:55:16.428427 ip-10-0-131-125 kubenswrapper[4521]: E1030 20:55:16.428392    4521 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lkn2k" containerName="extractor"
Oct 30 20:55:42.761048 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:55:42.761009    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-vtzgz"
Oct 30 20:55:42.810525 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:55:42.810484    4521 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-kmh84"]
Oct 30 20:55:42.810825 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:55:42.810769    4521 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-kmh84" podUID="eb9ee8df-9133-41fe-8128-cb1c28486d6b" containerName="manager" containerID="cri-o://6daaed1f41d3b0a7db4af093b8795ea1c92f2461889f85e2c42f2a82c3710259" gracePeriod=30
Oct 30 20:55:43.059939 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:55:43.059911    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-kmh84"
Oct 30 20:55:43.089369 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:55:43.089339    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb9ee8df-9133-41fe-8128-cb1c28486d6b-cert\" (UniqueName: \"kubernetes.io/secret/eb9ee8df-9133-41fe-8128-cb1c28486d6b-cert\") pod \"eb9ee8df-9133-41fe-8128-cb1c28486d6b\" (UID: \"eb9ee8df-9133-41fe-8128-cb1c28486d6b\") "
Oct 30 20:55:43.089548 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:55:43.089428    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb9ee8df-9133-41fe-8128-cb1c28486d6b-kube-api-access-w4p5x\" (UniqueName: \"kubernetes.io/projected/eb9ee8df-9133-41fe-8128-cb1c28486d6b-kube-api-access-w4p5x\") pod \"eb9ee8df-9133-41fe-8128-cb1c28486d6b\" (UID: \"eb9ee8df-9133-41fe-8128-cb1c28486d6b\") "
Oct 30 20:55:43.091764 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:55:43.091729    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb9ee8df-9133-41fe-8128-cb1c28486d6b-kube-api-access-w4p5x" pod "eb9ee8df-9133-41fe-8128-cb1c28486d6b" (UID: "eb9ee8df-9133-41fe-8128-cb1c28486d6b"). InnerVolumeSpecName "kube-api-access-w4p5x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:55:43.091764 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:55:43.091743    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb9ee8df-9133-41fe-8128-cb1c28486d6b-cert" pod "eb9ee8df-9133-41fe-8128-cb1c28486d6b" (UID: "eb9ee8df-9133-41fe-8128-cb1c28486d6b"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:55:43.189938 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:55:43.189898    4521 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4p5x\" (UniqueName: \"kubernetes.io/projected/eb9ee8df-9133-41fe-8128-cb1c28486d6b-kube-api-access-w4p5x\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:55:43.189938 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:55:43.189933    4521 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/eb9ee8df-9133-41fe-8128-cb1c28486d6b-cert\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:55:43.918268 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:55:43.918236    4521 generic.go:356] "Generic (PLEG): container finished" podID="eb9ee8df-9133-41fe-8128-cb1c28486d6b" containerID="6daaed1f41d3b0a7db4af093b8795ea1c92f2461889f85e2c42f2a82c3710259" exitCode=0
Oct 30 20:55:43.918706 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:55:43.918279    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-kmh84"
Oct 30 20:55:43.918706 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:55:43.918318    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-kmh84" event={"ID":"eb9ee8df-9133-41fe-8128-cb1c28486d6b","Type":"ContainerDied","Data":"6daaed1f41d3b0a7db4af093b8795ea1c92f2461889f85e2c42f2a82c3710259"}
Oct 30 20:55:43.918706 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:55:43.918387    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-kmh84" event={"ID":"eb9ee8df-9133-41fe-8128-cb1c28486d6b","Type":"ContainerDied","Data":"6f370fc5a08f4e9ee729544f0db6a147d88fc8912ecf52f1a5c22079bf00ba70"}
Oct 30 20:55:43.918706 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:55:43.918409    4521 scope.go:122] "RemoveContainer" containerID="6daaed1f41d3b0a7db4af093b8795ea1c92f2461889f85e2c42f2a82c3710259"
Oct 30 20:55:43.927498 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:55:43.927481    4521 scope.go:122] "RemoveContainer" containerID="6daaed1f41d3b0a7db4af093b8795ea1c92f2461889f85e2c42f2a82c3710259"
Oct 30 20:55:43.927744 ip-10-0-131-125 kubenswrapper[4521]: E1030 20:55:43.927717    4521 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6daaed1f41d3b0a7db4af093b8795ea1c92f2461889f85e2c42f2a82c3710259\": container with ID starting with 6daaed1f41d3b0a7db4af093b8795ea1c92f2461889f85e2c42f2a82c3710259 not found: ID does not exist" containerID="6daaed1f41d3b0a7db4af093b8795ea1c92f2461889f85e2c42f2a82c3710259"
Oct 30 20:55:43.927801 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:55:43.927745    4521 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6daaed1f41d3b0a7db4af093b8795ea1c92f2461889f85e2c42f2a82c3710259"} err="failed to get container status \"6daaed1f41d3b0a7db4af093b8795ea1c92f2461889f85e2c42f2a82c3710259\": rpc error: code = NotFound desc = could not find container \"6daaed1f41d3b0a7db4af093b8795ea1c92f2461889f85e2c42f2a82c3710259\": container with ID starting with 6daaed1f41d3b0a7db4af093b8795ea1c92f2461889f85e2c42f2a82c3710259 not found: ID does not exist"
Oct 30 20:55:43.944087 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:55:43.944061    4521 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-kmh84"]
Oct 30 20:55:43.946971 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:55:43.946951    4521 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-kmh84"]
Oct 30 20:55:44.790984 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:55:44.790952    4521 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb9ee8df-9133-41fe-8128-cb1c28486d6b" path="/var/lib/kubelet/pods/eb9ee8df-9133-41fe-8128-cb1c28486d6b/volumes"
Oct 30 20:55:46.430370 ip-10-0-131-125 kubenswrapper[4521]: E1030 20:55:46.430316    4521 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lkn2k" containerName="extractor"
Oct 30 20:55:59.891991 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:55:59.891957    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4kxb"]
Oct 30 20:56:05.211676 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:56:05.211583    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4kxb"]
Oct 30 20:56:06.918871 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:56:06.918832    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4kxb"]
Oct 30 20:56:10.501228 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:56:10.501180    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4kxb"]
Oct 30 20:56:14.733769 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:56:14.733723    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4kxb"]
Oct 30 20:56:16.431934 ip-10-0-131-125 kubenswrapper[4521]: E1030 20:56:16.431893    4521 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lkn2k" containerName="extractor"
Oct 30 20:56:18.575878 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:56:18.575838    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4kxb"]
Oct 30 20:56:39.292948 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:56:39.292914    4521 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-54dql"]
Oct 30 20:56:39.293408 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:56:39.293259    4521 cpu_manager.go:414] "containerMap: removing container" podUID="eb9ee8df-9133-41fe-8128-cb1c28486d6b" containerName="manager"
Oct 30 20:56:39.293408 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:56:39.293270    4521 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb9ee8df-9133-41fe-8128-cb1c28486d6b" containerName="manager"
Oct 30 20:56:39.293408 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:56:39.293359    4521 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb9ee8df-9133-41fe-8128-cb1c28486d6b" containerName="manager"
Oct 30 20:56:39.296976 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:56:39.296953    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-54dql"
Oct 30 20:56:39.310168 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:56:39.310137    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-54dql"]
Oct 30 20:56:39.382093 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:56:39.382052    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be342bb0-ebf7-4bb3-a929-5abfc455531e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-54dql\" (UID: \"be342bb0-ebf7-4bb3-a929-5abfc455531e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-54dql"
Oct 30 20:56:39.382093 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:56:39.382096    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be342bb0-ebf7-4bb3-a929-5abfc455531e-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-54dql\" (UID: \"be342bb0-ebf7-4bb3-a929-5abfc455531e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-54dql"
Oct 30 20:56:39.382312 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:56:39.382119    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be342bb0-ebf7-4bb3-a929-5abfc455531e-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-54dql\" (UID: \"be342bb0-ebf7-4bb3-a929-5abfc455531e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-54dql"
Oct 30 20:56:39.382312 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:56:39.382219    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be342bb0-ebf7-4bb3-a929-5abfc455531e-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-54dql\" (UID: \"be342bb0-ebf7-4bb3-a929-5abfc455531e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-54dql"
Oct 30 20:56:39.382312 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:56:39.382282    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be342bb0-ebf7-4bb3-a929-5abfc455531e-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-54dql\" (UID: \"be342bb0-ebf7-4bb3-a929-5abfc455531e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-54dql"
Oct 30 20:56:39.382446 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:56:39.382383    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xh4d2\" (UniqueName: \"kubernetes.io/projected/be342bb0-ebf7-4bb3-a929-5abfc455531e-kube-api-access-xh4d2\") pod \"maas-default-gateway-openshift-default-58c6585447-54dql\" (UID: \"be342bb0-ebf7-4bb3-a929-5abfc455531e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-54dql"
Oct 30 20:56:39.382446 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:56:39.382419    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be342bb0-ebf7-4bb3-a929-5abfc455531e-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-54dql\" (UID: \"be342bb0-ebf7-4bb3-a929-5abfc455531e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-54dql"
Oct 30 20:56:39.382536 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:56:39.382444    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be342bb0-ebf7-4bb3-a929-5abfc455531e-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-54dql\" (UID: \"be342bb0-ebf7-4bb3-a929-5abfc455531e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-54dql"
Oct 30 20:56:39.382579 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:56:39.382536    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be342bb0-ebf7-4bb3-a929-5abfc455531e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-54dql\" (UID: \"be342bb0-ebf7-4bb3-a929-5abfc455531e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-54dql"
Oct 30 20:56:39.483340 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:56:39.483293    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be342bb0-ebf7-4bb3-a929-5abfc455531e-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-54dql\" (UID: \"be342bb0-ebf7-4bb3-a929-5abfc455531e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-54dql"
Oct 30 20:56:39.483533 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:56:39.483366    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be342bb0-ebf7-4bb3-a929-5abfc455531e-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-54dql\" (UID: \"be342bb0-ebf7-4bb3-a929-5abfc455531e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-54dql"
Oct 30 20:56:39.483533 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:56:39.483390    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be342bb0-ebf7-4bb3-a929-5abfc455531e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-54dql\" (UID: \"be342bb0-ebf7-4bb3-a929-5abfc455531e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-54dql"
Oct 30 20:56:39.483533 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:56:39.483418    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be342bb0-ebf7-4bb3-a929-5abfc455531e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-54dql\" (UID: \"be342bb0-ebf7-4bb3-a929-5abfc455531e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-54dql"
Oct 30 20:56:39.483533 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:56:39.483434    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be342bb0-ebf7-4bb3-a929-5abfc455531e-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-54dql\" (UID: \"be342bb0-ebf7-4bb3-a929-5abfc455531e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-54dql"
Oct 30 20:56:39.483735 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:56:39.483606    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be342bb0-ebf7-4bb3-a929-5abfc455531e-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-54dql\" (UID: \"be342bb0-ebf7-4bb3-a929-5abfc455531e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-54dql"
Oct 30 20:56:39.483735 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:56:39.483665    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be342bb0-ebf7-4bb3-a929-5abfc455531e-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-54dql\" (UID: \"be342bb0-ebf7-4bb3-a929-5abfc455531e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-54dql"
Oct 30 20:56:39.483866 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:56:39.483744    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be342bb0-ebf7-4bb3-a929-5abfc455531e-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-54dql\" (UID: \"be342bb0-ebf7-4bb3-a929-5abfc455531e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-54dql"
Oct 30 20:56:39.483866 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:56:39.483797    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xh4d2\" (UniqueName: \"kubernetes.io/projected/be342bb0-ebf7-4bb3-a929-5abfc455531e-kube-api-access-xh4d2\") pod \"maas-default-gateway-openshift-default-58c6585447-54dql\" (UID: \"be342bb0-ebf7-4bb3-a929-5abfc455531e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-54dql"
Oct 30 20:56:39.483866 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:56:39.483853    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be342bb0-ebf7-4bb3-a929-5abfc455531e-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-54dql\" (UID: \"be342bb0-ebf7-4bb3-a929-5abfc455531e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-54dql"
Oct 30 20:56:39.484021 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:56:39.483948    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be342bb0-ebf7-4bb3-a929-5abfc455531e-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-54dql\" (UID: \"be342bb0-ebf7-4bb3-a929-5abfc455531e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-54dql"
Oct 30 20:56:39.484088 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:56:39.484043    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be342bb0-ebf7-4bb3-a929-5abfc455531e-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-54dql\" (UID: \"be342bb0-ebf7-4bb3-a929-5abfc455531e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-54dql"
Oct 30 20:56:39.484146 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:56:39.484101    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be342bb0-ebf7-4bb3-a929-5abfc455531e-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-54dql\" (UID: \"be342bb0-ebf7-4bb3-a929-5abfc455531e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-54dql"
Oct 30 20:56:39.484242 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:56:39.484220    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be342bb0-ebf7-4bb3-a929-5abfc455531e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-54dql\" (UID: \"be342bb0-ebf7-4bb3-a929-5abfc455531e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-54dql"
Oct 30 20:56:39.485747 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:56:39.485725    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be342bb0-ebf7-4bb3-a929-5abfc455531e-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-54dql\" (UID: \"be342bb0-ebf7-4bb3-a929-5abfc455531e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-54dql"
Oct 30 20:56:39.485968 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:56:39.485950    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be342bb0-ebf7-4bb3-a929-5abfc455531e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-54dql\" (UID: \"be342bb0-ebf7-4bb3-a929-5abfc455531e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-54dql"
Oct 30 20:56:39.497489 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:56:39.497458    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be342bb0-ebf7-4bb3-a929-5abfc455531e-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-54dql\" (UID: \"be342bb0-ebf7-4bb3-a929-5abfc455531e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-54dql"
Oct 30 20:56:39.497582 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:56:39.497548    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xh4d2\" (UniqueName: \"kubernetes.io/projected/be342bb0-ebf7-4bb3-a929-5abfc455531e-kube-api-access-xh4d2\") pod \"maas-default-gateway-openshift-default-58c6585447-54dql\" (UID: \"be342bb0-ebf7-4bb3-a929-5abfc455531e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-54dql"
Oct 30 20:56:39.613452 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:56:39.613365    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-54dql"
Oct 30 20:56:39.897716 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:56:39.897689    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-54dql"]
Oct 30 20:56:39.899398 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:56:39.899371    4521 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe342bb0_ebf7_4bb3_a929_5abfc455531e.slice/crio-0f105deb16b2be962734237f2dddbd2b0e5a5d2533aadfb0d9ed6038e133c358 WatchSource:0}: Error finding container 0f105deb16b2be962734237f2dddbd2b0e5a5d2533aadfb0d9ed6038e133c358: Status 404 returned error can't find the container with id 0f105deb16b2be962734237f2dddbd2b0e5a5d2533aadfb0d9ed6038e133c358
Oct 30 20:56:39.901430 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:56:39.901400    4521 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:56:39.901491 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:56:39.901467    4521 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:56:39.901527 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:56:39.901496    4521 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:56:40.211247 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:56:40.211209    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-54dql" event={"ID":"be342bb0-ebf7-4bb3-a929-5abfc455531e","Type":"ContainerStarted","Data":"ad973049c03a72fe30ab539571ece3e2307fd57f74e16ecc8dfb1a5eb2194e13"}
Oct 30 20:56:40.211444 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:56:40.211264    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-54dql" event={"ID":"be342bb0-ebf7-4bb3-a929-5abfc455531e","Type":"ContainerStarted","Data":"0f105deb16b2be962734237f2dddbd2b0e5a5d2533aadfb0d9ed6038e133c358"}
Oct 30 20:56:40.237884 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:56:40.237835    4521 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-54dql" podStartSLOduration=1.237818709 podStartE2EDuration="1.237818709s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:56:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:56:40.236979184 +0000 UTC m=+1067.582600423" watchObservedRunningTime="2026-10-30 20:56:40.237818709 +0000 UTC m=+1067.583439946"
Oct 30 20:56:40.627409 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:56:40.621717    4521 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-54dql"
Oct 30 20:56:40.634350 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:56:40.632454    4521 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-54dql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Oct 30 20:56:40.634350 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:56:40.632520    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-54dql" podUID="be342bb0-ebf7-4bb3-a929-5abfc455531e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Oct 30 20:56:41.622776 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:56:41.622738    4521 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-54dql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Oct 30 20:56:41.623096 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:56:41.623061    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-54dql" podUID="be342bb0-ebf7-4bb3-a929-5abfc455531e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Oct 30 20:56:42.631348 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:56:42.621728    4521 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-54dql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Oct 30 20:56:42.631348 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:56:42.621790    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-54dql" podUID="be342bb0-ebf7-4bb3-a929-5abfc455531e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Oct 30 20:56:43.629347 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:56:43.618704    4521 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-54dql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Oct 30 20:56:43.629347 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:56:43.618769    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-54dql" podUID="be342bb0-ebf7-4bb3-a929-5abfc455531e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Oct 30 20:56:44.628353 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:56:44.623765    4521 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-54dql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Oct 30 20:56:44.628353 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:56:44.623834    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-54dql" podUID="be342bb0-ebf7-4bb3-a929-5abfc455531e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Oct 30 20:56:45.627348 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:56:45.626735    4521 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-54dql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Oct 30 20:56:45.627348 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:56:45.626804    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-54dql" podUID="be342bb0-ebf7-4bb3-a929-5abfc455531e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Oct 30 20:56:46.429541 ip-10-0-131-125 kubenswrapper[4521]: E1030 20:56:46.429504    4521 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lkn2k" containerName="extractor"
Oct 30 20:56:46.614135 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:56:46.614101    4521 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-54dql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Oct 30 20:56:46.614305 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:56:46.614152    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-54dql" podUID="be342bb0-ebf7-4bb3-a929-5abfc455531e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Oct 30 20:56:48.082255 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:56:48.082220    4521 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-54dql"
Oct 30 20:56:48.082775 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:56:48.082618    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-54dql"
Oct 30 20:56:48.083369 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:56:48.083352    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-54dql"
Oct 30 20:56:54.349643 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:56:54.349609    4521 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-sq68x"]
Oct 30 20:56:54.353025 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:56:54.353001    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-sq68x"
Oct 30 20:56:54.377014 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:56:54.376983    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-sq68x"]
Oct 30 20:56:54.431156 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:56:54.431122    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16e33702-94a7-4c37-b430-1ac4f002832b-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-sq68x\" (UID: \"16e33702-94a7-4c37-b430-1ac4f002832b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-sq68x"
Oct 30 20:56:54.431335 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:56:54.431162    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhqf5\" (UniqueName: \"kubernetes.io/projected/16e33702-94a7-4c37-b430-1ac4f002832b-kube-api-access-vhqf5\") pod \"maas-default-gateway-openshift-default-58c6585447-sq68x\" (UID: \"16e33702-94a7-4c37-b430-1ac4f002832b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-sq68x"
Oct 30 20:56:54.431335 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:56:54.431258    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16e33702-94a7-4c37-b430-1ac4f002832b-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-sq68x\" (UID: \"16e33702-94a7-4c37-b430-1ac4f002832b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-sq68x"
Oct 30 20:56:54.431335 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:56:54.431286    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16e33702-94a7-4c37-b430-1ac4f002832b-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-sq68x\" (UID: \"16e33702-94a7-4c37-b430-1ac4f002832b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-sq68x"
Oct 30 20:56:54.431444 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:56:54.431351    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16e33702-94a7-4c37-b430-1ac4f002832b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-sq68x\" (UID: \"16e33702-94a7-4c37-b430-1ac4f002832b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-sq68x"
Oct 30 20:56:54.431444 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:56:54.431380    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16e33702-94a7-4c37-b430-1ac4f002832b-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-sq68x\" (UID: \"16e33702-94a7-4c37-b430-1ac4f002832b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-sq68x"
Oct 30 20:56:54.431444 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:56:54.431402    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16e33702-94a7-4c37-b430-1ac4f002832b-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-sq68x\" (UID: \"16e33702-94a7-4c37-b430-1ac4f002832b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-sq68x"
Oct 30 20:56:54.431444 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:56:54.431424    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16e33702-94a7-4c37-b430-1ac4f002832b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-sq68x\" (UID: \"16e33702-94a7-4c37-b430-1ac4f002832b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-sq68x"
Oct 30 20:56:54.431564 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:56:54.431479    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16e33702-94a7-4c37-b430-1ac4f002832b-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-sq68x\" (UID: \"16e33702-94a7-4c37-b430-1ac4f002832b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-sq68x"
Oct 30 20:56:54.532725 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:56:54.532686    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16e33702-94a7-4c37-b430-1ac4f002832b-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-sq68x\" (UID: \"16e33702-94a7-4c37-b430-1ac4f002832b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-sq68x"
Oct 30 20:56:54.532725 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:56:54.532727    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhqf5\" (UniqueName: \"kubernetes.io/projected/16e33702-94a7-4c37-b430-1ac4f002832b-kube-api-access-vhqf5\") pod \"maas-default-gateway-openshift-default-58c6585447-sq68x\" (UID: \"16e33702-94a7-4c37-b430-1ac4f002832b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-sq68x"
Oct 30 20:56:54.532942 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:56:54.532849    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16e33702-94a7-4c37-b430-1ac4f002832b-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-sq68x\" (UID: \"16e33702-94a7-4c37-b430-1ac4f002832b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-sq68x"
Oct 30 20:56:54.532942 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:56:54.532896    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16e33702-94a7-4c37-b430-1ac4f002832b-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-sq68x\" (UID: \"16e33702-94a7-4c37-b430-1ac4f002832b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-sq68x"
Oct 30 20:56:54.533040 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:56:54.532937    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16e33702-94a7-4c37-b430-1ac4f002832b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-sq68x\" (UID: \"16e33702-94a7-4c37-b430-1ac4f002832b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-sq68x"
Oct 30 20:56:54.533040 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:56:54.532980    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16e33702-94a7-4c37-b430-1ac4f002832b-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-sq68x\" (UID: \"16e33702-94a7-4c37-b430-1ac4f002832b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-sq68x"
Oct 30 20:56:54.533040 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:56:54.533013    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16e33702-94a7-4c37-b430-1ac4f002832b-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-sq68x\" (UID: \"16e33702-94a7-4c37-b430-1ac4f002832b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-sq68x"
Oct 30 20:56:54.533185 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:56:54.533048    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16e33702-94a7-4c37-b430-1ac4f002832b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-sq68x\" (UID: \"16e33702-94a7-4c37-b430-1ac4f002832b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-sq68x"
Oct 30 20:56:54.533185 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:56:54.533098    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16e33702-94a7-4c37-b430-1ac4f002832b-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-sq68x\" (UID: \"16e33702-94a7-4c37-b430-1ac4f002832b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-sq68x"
Oct 30 20:56:54.533185 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:56:54.533158    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16e33702-94a7-4c37-b430-1ac4f002832b-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-sq68x\" (UID: \"16e33702-94a7-4c37-b430-1ac4f002832b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-sq68x"
Oct 30 20:56:54.533359 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:56:54.533232    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16e33702-94a7-4c37-b430-1ac4f002832b-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-sq68x\" (UID: \"16e33702-94a7-4c37-b430-1ac4f002832b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-sq68x"
Oct 30 20:56:54.533421 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:56:54.533366    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16e33702-94a7-4c37-b430-1ac4f002832b-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-sq68x\" (UID: \"16e33702-94a7-4c37-b430-1ac4f002832b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-sq68x"
Oct 30 20:56:54.533496 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:56:54.533475    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16e33702-94a7-4c37-b430-1ac4f002832b-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-sq68x\" (UID: \"16e33702-94a7-4c37-b430-1ac4f002832b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-sq68x"
Oct 30 20:56:54.533783 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:56:54.533759    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16e33702-94a7-4c37-b430-1ac4f002832b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-sq68x\" (UID: \"16e33702-94a7-4c37-b430-1ac4f002832b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-sq68x"
Oct 30 20:56:54.535413 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:56:54.535384    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16e33702-94a7-4c37-b430-1ac4f002832b-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-sq68x\" (UID: \"16e33702-94a7-4c37-b430-1ac4f002832b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-sq68x"
Oct 30 20:56:54.535566 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:56:54.535552    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16e33702-94a7-4c37-b430-1ac4f002832b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-sq68x\" (UID: \"16e33702-94a7-4c37-b430-1ac4f002832b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-sq68x"
Oct 30 20:56:54.545626 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:56:54.545600    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhqf5\" (UniqueName: \"kubernetes.io/projected/16e33702-94a7-4c37-b430-1ac4f002832b-kube-api-access-vhqf5\") pod \"maas-default-gateway-openshift-default-58c6585447-sq68x\" (UID: \"16e33702-94a7-4c37-b430-1ac4f002832b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-sq68x"
Oct 30 20:56:54.546023 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:56:54.546007    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16e33702-94a7-4c37-b430-1ac4f002832b-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-sq68x\" (UID: \"16e33702-94a7-4c37-b430-1ac4f002832b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-sq68x"
Oct 30 20:56:54.665357 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:56:54.665240    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-sq68x"
Oct 30 20:56:54.952675 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:56:54.952646    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-sq68x"]
Oct 30 20:56:54.954000 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:56:54.953975    4521 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16e33702_94a7_4c37_b430_1ac4f002832b.slice/crio-2872e5bb18477333a50df4e0c546e4de1fa65b592daa6039c94435059cccbc08 WatchSource:0}: Error finding container 2872e5bb18477333a50df4e0c546e4de1fa65b592daa6039c94435059cccbc08: Status 404 returned error can't find the container with id 2872e5bb18477333a50df4e0c546e4de1fa65b592daa6039c94435059cccbc08
Oct 30 20:56:54.956185 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:56:54.956152    4521 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:56:54.956276 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:56:54.956230    4521 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:56:54.956354 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:56:54.956276    4521 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:56:55.309633 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:56:55.309546    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-sq68x" event={"ID":"16e33702-94a7-4c37-b430-1ac4f002832b","Type":"ContainerStarted","Data":"7c80e51bd8781a3c0b20d8fa3f7b9051d5c61245c2928324f61e13ad1d9f0cb3"}
Oct 30 20:56:55.309633 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:56:55.309603    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-sq68x" event={"ID":"16e33702-94a7-4c37-b430-1ac4f002832b","Type":"ContainerStarted","Data":"2872e5bb18477333a50df4e0c546e4de1fa65b592daa6039c94435059cccbc08"}
Oct 30 20:56:55.347615 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:56:55.345923    4521 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-sq68x" podStartSLOduration=1.345902956 podStartE2EDuration="1.345902956s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:56:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:56:55.340510662 +0000 UTC m=+1082.686131899" watchObservedRunningTime="2026-10-30 20:56:55.345902956 +0000 UTC m=+1082.691524194"
Oct 30 20:56:55.666392 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:56:55.666300    4521 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-sq68x"
Oct 30 20:56:56.667236 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:56:56.667201    4521 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-sq68x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Oct 30 20:56:56.667646 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:56:56.667254    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-sq68x" podUID="16e33702-94a7-4c37-b430-1ac4f002832b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Oct 30 20:56:57.666100 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:56:57.666066    4521 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-sq68x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Oct 30 20:56:57.666278 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:56:57.666121    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-sq68x" podUID="16e33702-94a7-4c37-b430-1ac4f002832b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Oct 30 20:56:58.665802 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:56:58.665767    4521 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-sq68x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Oct 30 20:56:58.666203 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:56:58.665824    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-sq68x" podUID="16e33702-94a7-4c37-b430-1ac4f002832b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Oct 30 20:56:59.666605 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:56:59.666567    4521 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-sq68x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Oct 30 20:56:59.667092 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:56:59.666631    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-sq68x" podUID="16e33702-94a7-4c37-b430-1ac4f002832b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Oct 30 20:57:00.665889 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:00.665854    4521 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-sq68x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Oct 30 20:57:00.666074 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:00.665909    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-sq68x" podUID="16e33702-94a7-4c37-b430-1ac4f002832b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Oct 30 20:57:01.666492 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:01.666451    4521 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-sq68x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Oct 30 20:57:01.666973 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:01.666533    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-sq68x" podUID="16e33702-94a7-4c37-b430-1ac4f002832b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Oct 30 20:57:03.139712 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:03.139661    4521 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-sq68x"
Oct 30 20:57:03.140226 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:03.140056    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-sq68x"
Oct 30 20:57:03.141237 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:03.141217    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-sq68x"
Oct 30 20:57:16.428334 ip-10-0-131-125 kubenswrapper[4521]: E1030 20:57:16.428285    4521 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lkn2k" containerName="extractor"
Oct 30 20:57:33.239121 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:33.239084    4521 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-c4b379-7c594b8495-t7ncb"]
Oct 30 20:57:33.243280 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:33.243254    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c4b379-7c594b8495-t7ncb"
Oct 30 20:57:33.252376 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:33.252356    4521 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-c4b379\""
Oct 30 20:57:33.263304 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:33.263274    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-c4b379-7c594b8495-t7ncb"]
Oct 30 20:57:33.265554 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:33.265526    4521 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-c4b379-779cc77c6-b8gqg"]
Oct 30 20:57:33.269154 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:33.269133    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-c4b379-779cc77c6-b8gqg"
Oct 30 20:57:33.278644 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:33.278624    4521 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-c4b379-dockercfg-26gln\""
Oct 30 20:57:33.302174 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:33.302146    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-c4b379-779cc77c6-b8gqg"]
Oct 30 20:57:33.398738 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:33.398706    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77f3a44b-d4f2-4506-b386-7a216ee118d9-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-c4b379-7c594b8495-t7ncb\" (UID: \"77f3a44b-d4f2-4506-b386-7a216ee118d9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c4b379-7c594b8495-t7ncb"
Oct 30 20:57:33.398899 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:33.398743    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6b86c\" (UniqueName: \"kubernetes.io/projected/c29ead58-26c1-4720-ac49-99b5e045b85d-kube-api-access-6b86c\") pod \"payload-processing-e2e-worker-w1-c4b379-779cc77c6-b8gqg\" (UID: \"c29ead58-26c1-4720-ac49-99b5e045b85d\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-c4b379-779cc77c6-b8gqg"
Oct 30 20:57:33.398899 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:33.398772    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c29ead58-26c1-4720-ac49-99b5e045b85d-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-c4b379-779cc77c6-b8gqg\" (UID: \"c29ead58-26c1-4720-ac49-99b5e045b85d\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-c4b379-779cc77c6-b8gqg"
Oct 30 20:57:33.398899 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:33.398817    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpwq5\" (UniqueName: \"kubernetes.io/projected/77f3a44b-d4f2-4506-b386-7a216ee118d9-kube-api-access-qpwq5\") pod \"payload-pre-processing-e2e-worker-w1-c4b379-7c594b8495-t7ncb\" (UID: \"77f3a44b-d4f2-4506-b386-7a216ee118d9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c4b379-7c594b8495-t7ncb"
Oct 30 20:57:33.504398 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:33.500429    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77f3a44b-d4f2-4506-b386-7a216ee118d9-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-c4b379-7c594b8495-t7ncb\" (UID: \"77f3a44b-d4f2-4506-b386-7a216ee118d9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c4b379-7c594b8495-t7ncb"
Oct 30 20:57:33.504398 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:33.500492    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6b86c\" (UniqueName: \"kubernetes.io/projected/c29ead58-26c1-4720-ac49-99b5e045b85d-kube-api-access-6b86c\") pod \"payload-processing-e2e-worker-w1-c4b379-779cc77c6-b8gqg\" (UID: \"c29ead58-26c1-4720-ac49-99b5e045b85d\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-c4b379-779cc77c6-b8gqg"
Oct 30 20:57:33.504398 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:33.500521    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c29ead58-26c1-4720-ac49-99b5e045b85d-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-c4b379-779cc77c6-b8gqg\" (UID: \"c29ead58-26c1-4720-ac49-99b5e045b85d\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-c4b379-779cc77c6-b8gqg"
Oct 30 20:57:33.504398 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:33.500544    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpwq5\" (UniqueName: \"kubernetes.io/projected/77f3a44b-d4f2-4506-b386-7a216ee118d9-kube-api-access-qpwq5\") pod \"payload-pre-processing-e2e-worker-w1-c4b379-7c594b8495-t7ncb\" (UID: \"77f3a44b-d4f2-4506-b386-7a216ee118d9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c4b379-7c594b8495-t7ncb"
Oct 30 20:57:33.504398 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:33.501841    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77f3a44b-d4f2-4506-b386-7a216ee118d9-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-c4b379-7c594b8495-t7ncb\" (UID: \"77f3a44b-d4f2-4506-b386-7a216ee118d9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c4b379-7c594b8495-t7ncb"
Oct 30 20:57:33.504398 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:33.502812    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c29ead58-26c1-4720-ac49-99b5e045b85d-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-c4b379-779cc77c6-b8gqg\" (UID: \"c29ead58-26c1-4720-ac49-99b5e045b85d\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-c4b379-779cc77c6-b8gqg"
Oct 30 20:57:33.520656 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:33.520612    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpwq5\" (UniqueName: \"kubernetes.io/projected/77f3a44b-d4f2-4506-b386-7a216ee118d9-kube-api-access-qpwq5\") pod \"payload-pre-processing-e2e-worker-w1-c4b379-7c594b8495-t7ncb\" (UID: \"77f3a44b-d4f2-4506-b386-7a216ee118d9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c4b379-7c594b8495-t7ncb"
Oct 30 20:57:33.520831 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:33.520808    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6b86c\" (UniqueName: \"kubernetes.io/projected/c29ead58-26c1-4720-ac49-99b5e045b85d-kube-api-access-6b86c\") pod \"payload-processing-e2e-worker-w1-c4b379-779cc77c6-b8gqg\" (UID: \"c29ead58-26c1-4720-ac49-99b5e045b85d\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-c4b379-779cc77c6-b8gqg"
Oct 30 20:57:33.557374 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:33.557316    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c4b379-7c594b8495-t7ncb"
Oct 30 20:57:33.584830 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:33.584800    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-c4b379-779cc77c6-b8gqg"
Oct 30 20:57:33.699273 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:33.699235    4521 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-39fb96-7954dbb4bc-r8z6k"]
Oct 30 20:57:33.704185 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:33.704161    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-39fb96-7954dbb4bc-r8z6k"
Oct 30 20:57:33.749257 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:33.749219    4521 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-39fb96\""
Oct 30 20:57:33.750676 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:33.750642    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-39fb96-7954dbb4bc-r8z6k"]
Oct 30 20:57:33.806415 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:33.806280    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dv47\" (UniqueName: \"kubernetes.io/projected/f3530279-92bb-445f-95b2-441c22fadc9f-kube-api-access-9dv47\") pod \"payload-pre-processing-e2e-worker-w6-39fb96-7954dbb4bc-r8z6k\" (UID: \"f3530279-92bb-445f-95b2-441c22fadc9f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-39fb96-7954dbb4bc-r8z6k"
Oct 30 20:57:33.806592 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:33.806521    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3530279-92bb-445f-95b2-441c22fadc9f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-39fb96-7954dbb4bc-r8z6k\" (UID: \"f3530279-92bb-445f-95b2-441c22fadc9f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-39fb96-7954dbb4bc-r8z6k"
Oct 30 20:57:33.886031 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:33.886003    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-c4b379-7c594b8495-t7ncb"]
Oct 30 20:57:33.888447 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:57:33.888415    4521 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77f3a44b_d4f2_4506_b386_7a216ee118d9.slice/crio-a7fd6974ffd5be56409d8b5a3e664da90b6e2633a0b9e37d8b58b7ec8eab88a0 WatchSource:0}: Error finding container a7fd6974ffd5be56409d8b5a3e664da90b6e2633a0b9e37d8b58b7ec8eab88a0: Status 404 returned error can't find the container with id a7fd6974ffd5be56409d8b5a3e664da90b6e2633a0b9e37d8b58b7ec8eab88a0
Oct 30 20:57:33.907860 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:33.907835    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3530279-92bb-445f-95b2-441c22fadc9f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-39fb96-7954dbb4bc-r8z6k\" (UID: \"f3530279-92bb-445f-95b2-441c22fadc9f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-39fb96-7954dbb4bc-r8z6k"
Oct 30 20:57:33.908274 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:33.907958    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dv47\" (UniqueName: \"kubernetes.io/projected/f3530279-92bb-445f-95b2-441c22fadc9f-kube-api-access-9dv47\") pod \"payload-pre-processing-e2e-worker-w6-39fb96-7954dbb4bc-r8z6k\" (UID: \"f3530279-92bb-445f-95b2-441c22fadc9f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-39fb96-7954dbb4bc-r8z6k"
Oct 30 20:57:33.908411 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:33.908356    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-c4b379-779cc77c6-b8gqg"]
Oct 30 20:57:33.908723 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:33.908692    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3530279-92bb-445f-95b2-441c22fadc9f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-39fb96-7954dbb4bc-r8z6k\" (UID: \"f3530279-92bb-445f-95b2-441c22fadc9f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-39fb96-7954dbb4bc-r8z6k"
Oct 30 20:57:33.910354 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:57:33.910306    4521 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc29ead58_26c1_4720_ac49_99b5e045b85d.slice/crio-b969b1211f7c59b209365e82d4808d23da178eb90b70162032497b573256213b WatchSource:0}: Error finding container b969b1211f7c59b209365e82d4808d23da178eb90b70162032497b573256213b: Status 404 returned error can't find the container with id b969b1211f7c59b209365e82d4808d23da178eb90b70162032497b573256213b
Oct 30 20:57:33.922280 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:33.922254    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dv47\" (UniqueName: \"kubernetes.io/projected/f3530279-92bb-445f-95b2-441c22fadc9f-kube-api-access-9dv47\") pod \"payload-pre-processing-e2e-worker-w6-39fb96-7954dbb4bc-r8z6k\" (UID: \"f3530279-92bb-445f-95b2-441c22fadc9f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-39fb96-7954dbb4bc-r8z6k"
Oct 30 20:57:34.021762 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:34.021719    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-39fb96-7954dbb4bc-r8z6k"
Oct 30 20:57:34.312564 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:34.312531    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-39fb96-7954dbb4bc-r8z6k"]
Oct 30 20:57:34.314301 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:57:34.314276    4521 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3530279_92bb_445f_95b2_441c22fadc9f.slice/crio-b8a1b2d998fd5eaa5f4db5ca8125dd56822b6823a4bda8cf0af600434a781d17 WatchSource:0}: Error finding container b8a1b2d998fd5eaa5f4db5ca8125dd56822b6823a4bda8cf0af600434a781d17: Status 404 returned error can't find the container with id b8a1b2d998fd5eaa5f4db5ca8125dd56822b6823a4bda8cf0af600434a781d17
Oct 30 20:57:34.555539 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:34.555499    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c4b379-7c594b8495-t7ncb" event={"ID":"77f3a44b-d4f2-4506-b386-7a216ee118d9","Type":"ContainerStarted","Data":"a7fd6974ffd5be56409d8b5a3e664da90b6e2633a0b9e37d8b58b7ec8eab88a0"}
Oct 30 20:57:34.557941 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:34.557873    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-39fb96-7954dbb4bc-r8z6k" event={"ID":"f3530279-92bb-445f-95b2-441c22fadc9f","Type":"ContainerStarted","Data":"b8a1b2d998fd5eaa5f4db5ca8125dd56822b6823a4bda8cf0af600434a781d17"}
Oct 30 20:57:34.559375 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:34.559333    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-c4b379-779cc77c6-b8gqg" event={"ID":"c29ead58-26c1-4720-ac49-99b5e045b85d","Type":"ContainerStarted","Data":"b969b1211f7c59b209365e82d4808d23da178eb90b70162032497b573256213b"}
Oct 30 20:57:35.641267 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:35.641226    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4kxb"]
Oct 30 20:57:37.584729 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:37.584690    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c4b379-7c594b8495-t7ncb" event={"ID":"77f3a44b-d4f2-4506-b386-7a216ee118d9","Type":"ContainerStarted","Data":"9fb1f2ff6267618504e29c1f8c48c620c26f736f88391c462f6c660047b457e1"}
Oct 30 20:57:37.585227 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:37.584767    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c4b379-7c594b8495-t7ncb"
Oct 30 20:57:37.586496 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:37.586469    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-39fb96-7954dbb4bc-r8z6k" event={"ID":"f3530279-92bb-445f-95b2-441c22fadc9f","Type":"ContainerStarted","Data":"57592b57782433ff180994a4f8417e3267dac98c3d770d97e6aad2722facb942"}
Oct 30 20:57:37.586640 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:37.586536    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-39fb96-7954dbb4bc-r8z6k"
Oct 30 20:57:37.588108 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:37.588086    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-c4b379-779cc77c6-b8gqg" event={"ID":"c29ead58-26c1-4720-ac49-99b5e045b85d","Type":"ContainerStarted","Data":"9ff6b4c9dad5e2f78b264f46d6a0af55d026adc75fb3ec343f9e03c2eb415d3e"}
Oct 30 20:57:37.588209 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:37.588194    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-c4b379-779cc77c6-b8gqg"
Oct 30 20:57:37.604300 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:37.604260    4521 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c4b379-7c594b8495-t7ncb" podStartSLOduration=1.357089708 podStartE2EDuration="4.604247832s" totalImagesPullingTime="3.247158124s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:57:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:57:37.602880855 +0000 UTC m=+1124.948502094" watchObservedRunningTime="2026-10-30 20:57:37.604247832 +0000 UTC m=+1124.949869076"
Oct 30 20:57:37.658082 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:37.658020    4521 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-c4b379-779cc77c6-b8gqg" podStartSLOduration=1.375325983 podStartE2EDuration="4.657976644s" totalImagesPullingTime="3.282650661s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:57:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:57:37.628700761 +0000 UTC m=+1124.974322017" watchObservedRunningTime="2026-10-30 20:57:37.657976644 +0000 UTC m=+1125.003597883"
Oct 30 20:57:37.658619 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:37.658587    4521 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-39fb96-7954dbb4bc-r8z6k" podStartSLOduration=1.7472410040000002 podStartE2EDuration="4.658579668s" totalImagesPullingTime="2.911338664s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:57:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:57:37.656808536 +0000 UTC m=+1125.002429774" watchObservedRunningTime="2026-10-30 20:57:37.658579668 +0000 UTC m=+1125.004200910"
Oct 30 20:57:38.776799 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:38.776766    4521 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-39fb96-759bc7c496-29tpq"]
Oct 30 20:57:38.780642 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:38.780622    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-39fb96-759bc7c496-29tpq"
Oct 30 20:57:38.785666 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:38.785639    4521 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w6-39fb96-serving-cert\""
Oct 30 20:57:38.791938 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:38.790506    4521 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:57:38.791938 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:38.790608    4521 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 30 20:57:38.791938 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:38.790679    4521 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-57jm5\""
Oct 30 20:57:38.791938 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:38.791421    4521 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:57:38.799591 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:38.799567    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-39fb96-759bc7c496-29tpq"]
Oct 30 20:57:38.864076 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:38.864039    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a9ccb4e-15bd-4fe3-8f85-45ab99eea028-maas-api-tls\") pod \"maas-api-e2e-worker-w6-39fb96-759bc7c496-29tpq\" (UID: \"9a9ccb4e-15bd-4fe3-8f85-45ab99eea028\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-39fb96-759bc7c496-29tpq"
Oct 30 20:57:38.864076 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:38.864079    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a9ccb4e-15bd-4fe3-8f85-45ab99eea028-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-39fb96-759bc7c496-29tpq\" (UID: \"9a9ccb4e-15bd-4fe3-8f85-45ab99eea028\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-39fb96-759bc7c496-29tpq"
Oct 30 20:57:38.864287 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:38.864260    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a9ccb4e-15bd-4fe3-8f85-45ab99eea028-metrics-certs\") pod \"maas-api-e2e-worker-w6-39fb96-759bc7c496-29tpq\" (UID: \"9a9ccb4e-15bd-4fe3-8f85-45ab99eea028\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-39fb96-759bc7c496-29tpq"
Oct 30 20:57:38.864287 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:38.864283    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hz8tl\" (UniqueName: \"kubernetes.io/projected/9a9ccb4e-15bd-4fe3-8f85-45ab99eea028-kube-api-access-hz8tl\") pod \"maas-api-e2e-worker-w6-39fb96-759bc7c496-29tpq\" (UID: \"9a9ccb4e-15bd-4fe3-8f85-45ab99eea028\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-39fb96-759bc7c496-29tpq"
Oct 30 20:57:38.965190 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:38.965151    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a9ccb4e-15bd-4fe3-8f85-45ab99eea028-maas-api-tls\") pod \"maas-api-e2e-worker-w6-39fb96-759bc7c496-29tpq\" (UID: \"9a9ccb4e-15bd-4fe3-8f85-45ab99eea028\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-39fb96-759bc7c496-29tpq"
Oct 30 20:57:38.965190 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:38.965189    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a9ccb4e-15bd-4fe3-8f85-45ab99eea028-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-39fb96-759bc7c496-29tpq\" (UID: \"9a9ccb4e-15bd-4fe3-8f85-45ab99eea028\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-39fb96-759bc7c496-29tpq"
Oct 30 20:57:38.965443 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:38.965229    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a9ccb4e-15bd-4fe3-8f85-45ab99eea028-metrics-certs\") pod \"maas-api-e2e-worker-w6-39fb96-759bc7c496-29tpq\" (UID: \"9a9ccb4e-15bd-4fe3-8f85-45ab99eea028\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-39fb96-759bc7c496-29tpq"
Oct 30 20:57:38.965443 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:38.965246    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hz8tl\" (UniqueName: \"kubernetes.io/projected/9a9ccb4e-15bd-4fe3-8f85-45ab99eea028-kube-api-access-hz8tl\") pod \"maas-api-e2e-worker-w6-39fb96-759bc7c496-29tpq\" (UID: \"9a9ccb4e-15bd-4fe3-8f85-45ab99eea028\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-39fb96-759bc7c496-29tpq"
Oct 30 20:57:38.965981 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:38.965957    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a9ccb4e-15bd-4fe3-8f85-45ab99eea028-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-39fb96-759bc7c496-29tpq\" (UID: \"9a9ccb4e-15bd-4fe3-8f85-45ab99eea028\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-39fb96-759bc7c496-29tpq"
Oct 30 20:57:38.967767 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:38.967740    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a9ccb4e-15bd-4fe3-8f85-45ab99eea028-metrics-certs\") pod \"maas-api-e2e-worker-w6-39fb96-759bc7c496-29tpq\" (UID: \"9a9ccb4e-15bd-4fe3-8f85-45ab99eea028\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-39fb96-759bc7c496-29tpq"
Oct 30 20:57:38.967899 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:38.967781    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a9ccb4e-15bd-4fe3-8f85-45ab99eea028-maas-api-tls\") pod \"maas-api-e2e-worker-w6-39fb96-759bc7c496-29tpq\" (UID: \"9a9ccb4e-15bd-4fe3-8f85-45ab99eea028\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-39fb96-759bc7c496-29tpq"
Oct 30 20:57:38.978546 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:38.978516    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hz8tl\" (UniqueName: \"kubernetes.io/projected/9a9ccb4e-15bd-4fe3-8f85-45ab99eea028-kube-api-access-hz8tl\") pod \"maas-api-e2e-worker-w6-39fb96-759bc7c496-29tpq\" (UID: \"9a9ccb4e-15bd-4fe3-8f85-45ab99eea028\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-39fb96-759bc7c496-29tpq"
Oct 30 20:57:39.098575 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:39.098477    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-39fb96-759bc7c496-29tpq"
Oct 30 20:57:39.383970 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:39.383943    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-39fb96-759bc7c496-29tpq"]
Oct 30 20:57:39.385163 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:57:39.385140    4521 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a9ccb4e_15bd_4fe3_8f85_45ab99eea028.slice/crio-6e1584435ac2130e63215b186ee004cb2b26a6565c919649918ee55b67737534 WatchSource:0}: Error finding container 6e1584435ac2130e63215b186ee004cb2b26a6565c919649918ee55b67737534: Status 404 returned error can't find the container with id 6e1584435ac2130e63215b186ee004cb2b26a6565c919649918ee55b67737534
Oct 30 20:57:39.600099 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:39.600063    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-39fb96-759bc7c496-29tpq" event={"ID":"9a9ccb4e-15bd-4fe3-8f85-45ab99eea028","Type":"ContainerStarted","Data":"6e1584435ac2130e63215b186ee004cb2b26a6565c919649918ee55b67737534"}
Oct 30 20:57:39.875598 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:39.875559    4521 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-65968b-6876d4dbfb-js2xm"]
Oct 30 20:57:39.879701 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:39.879674    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-65968b-6876d4dbfb-js2xm"
Oct 30 20:57:39.893704 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:39.888857    4521 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w5-65968b-serving-cert\""
Oct 30 20:57:39.899296 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:39.899224    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-65968b-6876d4dbfb-js2xm"]
Oct 30 20:57:39.931556 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:39.931518    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4kxb"]
Oct 30 20:57:39.979453 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:39.979417    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56bp7\" (UniqueName: \"kubernetes.io/projected/7cb73975-c1ac-4957-9337-6ada4d05a2dc-kube-api-access-56bp7\") pod \"maas-api-e2e-worker-w5-65968b-6876d4dbfb-js2xm\" (UID: \"7cb73975-c1ac-4957-9337-6ada4d05a2dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-65968b-6876d4dbfb-js2xm"
Oct 30 20:57:39.979680 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:39.979544    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7cb73975-c1ac-4957-9337-6ada4d05a2dc-maas-api-tls\") pod \"maas-api-e2e-worker-w5-65968b-6876d4dbfb-js2xm\" (UID: \"7cb73975-c1ac-4957-9337-6ada4d05a2dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-65968b-6876d4dbfb-js2xm"
Oct 30 20:57:39.979680 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:39.979576    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cb73975-c1ac-4957-9337-6ada4d05a2dc-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-65968b-6876d4dbfb-js2xm\" (UID: \"7cb73975-c1ac-4957-9337-6ada4d05a2dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-65968b-6876d4dbfb-js2xm"
Oct 30 20:57:39.979680 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:39.979612    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7cb73975-c1ac-4957-9337-6ada4d05a2dc-metrics-certs\") pod \"maas-api-e2e-worker-w5-65968b-6876d4dbfb-js2xm\" (UID: \"7cb73975-c1ac-4957-9337-6ada4d05a2dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-65968b-6876d4dbfb-js2xm"
Oct 30 20:57:40.081086 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:40.081049    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7cb73975-c1ac-4957-9337-6ada4d05a2dc-maas-api-tls\") pod \"maas-api-e2e-worker-w5-65968b-6876d4dbfb-js2xm\" (UID: \"7cb73975-c1ac-4957-9337-6ada4d05a2dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-65968b-6876d4dbfb-js2xm"
Oct 30 20:57:40.081274 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:40.081096    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cb73975-c1ac-4957-9337-6ada4d05a2dc-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-65968b-6876d4dbfb-js2xm\" (UID: \"7cb73975-c1ac-4957-9337-6ada4d05a2dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-65968b-6876d4dbfb-js2xm"
Oct 30 20:57:40.081274 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:40.081161    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7cb73975-c1ac-4957-9337-6ada4d05a2dc-metrics-certs\") pod \"maas-api-e2e-worker-w5-65968b-6876d4dbfb-js2xm\" (UID: \"7cb73975-c1ac-4957-9337-6ada4d05a2dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-65968b-6876d4dbfb-js2xm"
Oct 30 20:57:40.081274 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:40.081226    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56bp7\" (UniqueName: \"kubernetes.io/projected/7cb73975-c1ac-4957-9337-6ada4d05a2dc-kube-api-access-56bp7\") pod \"maas-api-e2e-worker-w5-65968b-6876d4dbfb-js2xm\" (UID: \"7cb73975-c1ac-4957-9337-6ada4d05a2dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-65968b-6876d4dbfb-js2xm"
Oct 30 20:57:40.082708 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:40.082680    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cb73975-c1ac-4957-9337-6ada4d05a2dc-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-65968b-6876d4dbfb-js2xm\" (UID: \"7cb73975-c1ac-4957-9337-6ada4d05a2dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-65968b-6876d4dbfb-js2xm"
Oct 30 20:57:40.085087 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:40.085061    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7cb73975-c1ac-4957-9337-6ada4d05a2dc-maas-api-tls\") pod \"maas-api-e2e-worker-w5-65968b-6876d4dbfb-js2xm\" (UID: \"7cb73975-c1ac-4957-9337-6ada4d05a2dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-65968b-6876d4dbfb-js2xm"
Oct 30 20:57:40.085743 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:40.085719    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7cb73975-c1ac-4957-9337-6ada4d05a2dc-metrics-certs\") pod \"maas-api-e2e-worker-w5-65968b-6876d4dbfb-js2xm\" (UID: \"7cb73975-c1ac-4957-9337-6ada4d05a2dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-65968b-6876d4dbfb-js2xm"
Oct 30 20:57:40.095776 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:40.095748    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56bp7\" (UniqueName: \"kubernetes.io/projected/7cb73975-c1ac-4957-9337-6ada4d05a2dc-kube-api-access-56bp7\") pod \"maas-api-e2e-worker-w5-65968b-6876d4dbfb-js2xm\" (UID: \"7cb73975-c1ac-4957-9337-6ada4d05a2dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-65968b-6876d4dbfb-js2xm"
Oct 30 20:57:40.200818 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:40.200749    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-65968b-6876d4dbfb-js2xm"
Oct 30 20:57:40.539679 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:40.539594    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-65968b-6876d4dbfb-js2xm"]
Oct 30 20:57:40.620385 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:40.614046    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-65968b-6876d4dbfb-js2xm" event={"ID":"7cb73975-c1ac-4957-9337-6ada4d05a2dc","Type":"ContainerStarted","Data":"52b6b7ba03575592d77d6ea5013927bf2423f6c41d112f30cfbe3396a9732293"}
Oct 30 20:57:43.638762 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:43.638711    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-65968b-6876d4dbfb-js2xm" event={"ID":"7cb73975-c1ac-4957-9337-6ada4d05a2dc","Type":"ContainerStarted","Data":"5fc3147ddba94cc576ec94aa37ad1ff19456d41f1bcff9c22589c1333e26c581"}
Oct 30 20:57:43.639267 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:43.638824    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-65968b-6876d4dbfb-js2xm"
Oct 30 20:57:43.640745 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:43.640719    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-39fb96-759bc7c496-29tpq" event={"ID":"9a9ccb4e-15bd-4fe3-8f85-45ab99eea028","Type":"ContainerStarted","Data":"411f92977fe45223c42ae4f05363d09a9d2c0760480fe50f22d1f1b7421158e9"}
Oct 30 20:57:43.640866 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:43.640804    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-39fb96-759bc7c496-29tpq"
Oct 30 20:57:43.659688 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:43.659637    4521 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-65968b-6876d4dbfb-js2xm" podStartSLOduration=2.555511622 podStartE2EDuration="4.65962503s" totalImagesPullingTime="2.104113408s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:57:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:57:43.658115371 +0000 UTC m=+1131.003736609" watchObservedRunningTime="2026-10-30 20:57:43.65962503 +0000 UTC m=+1131.005246265"
Oct 30 20:57:43.678877 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:43.678812    4521 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-39fb96-759bc7c496-29tpq" podStartSLOduration=2.42201703 podStartE2EDuration="5.678795327s" totalImagesPullingTime="3.256778297s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:57:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:57:43.678227959 +0000 UTC m=+1131.023849207" watchObservedRunningTime="2026-10-30 20:57:43.678795327 +0000 UTC m=+1131.024416567"
Oct 30 20:57:44.498854 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:44.498821    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4kxb"]
Oct 30 20:57:46.429707 ip-10-0-131-125 kubenswrapper[4521]: E1030 20:57:46.429668    4521 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lkn2k" containerName="extractor"
Oct 30 20:57:48.597172 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:48.597139    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-39fb96-7954dbb4bc-r8z6k"
Oct 30 20:57:48.597681 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:48.597213    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-c4b379-779cc77c6-b8gqg"
Oct 30 20:57:48.597681 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:48.597356    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c4b379-7c594b8495-t7ncb"
Oct 30 20:57:49.654711 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:49.654679    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-39fb96-759bc7c496-29tpq"
Oct 30 20:57:49.655244 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:49.655224    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-65968b-6876d4dbfb-js2xm"
Oct 30 20:57:53.603418 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:53.603384    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4kxb"]
Oct 30 20:57:53.726141 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:53.726097    4521 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-32a969/simulator-w2-32a969-kserve-6cc8dfc44c-dwhlt"]
Oct 30 20:57:53.731450 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:53.731424    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-32a969/simulator-w2-32a969-kserve-6cc8dfc44c-dwhlt"
Oct 30 20:57:53.754941 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:53.754907    4521 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-32a969\"/\"kube-root-ca.crt\""
Oct 30 20:57:53.755215 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:53.755196    4521 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-32a969\"/\"default-dockercfg-6s24m\""
Oct 30 20:57:53.755309 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:53.755098    4521 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-32a969\"/\"openshift-service-ca.crt\""
Oct 30 20:57:53.756302 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:53.756278    4521 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-32a969\"/\"simulator-w2-32a969-kserve-self-signed-certs\""
Oct 30 20:57:53.756458 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:53.756379    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-32a969/simulator-w2-32a969-kserve-6cc8dfc44c-dwhlt"]
Oct 30 20:57:53.836897 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:53.836851    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62a6adf7-d17a-4fec-a707-a2d549b67753-dshm\") pod \"simulator-w2-32a969-kserve-6cc8dfc44c-dwhlt\" (UID: \"62a6adf7-d17a-4fec-a707-a2d549b67753\") " pod="e2e-models-e2e-worker-w2-32a969/simulator-w2-32a969-kserve-6cc8dfc44c-dwhlt"
Oct 30 20:57:53.836897 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:53.836892    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62a6adf7-d17a-4fec-a707-a2d549b67753-home\") pod \"simulator-w2-32a969-kserve-6cc8dfc44c-dwhlt\" (UID: \"62a6adf7-d17a-4fec-a707-a2d549b67753\") " pod="e2e-models-e2e-worker-w2-32a969/simulator-w2-32a969-kserve-6cc8dfc44c-dwhlt"
Oct 30 20:57:53.837178 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:53.837032    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62a6adf7-d17a-4fec-a707-a2d549b67753-model-cache\") pod \"simulator-w2-32a969-kserve-6cc8dfc44c-dwhlt\" (UID: \"62a6adf7-d17a-4fec-a707-a2d549b67753\") " pod="e2e-models-e2e-worker-w2-32a969/simulator-w2-32a969-kserve-6cc8dfc44c-dwhlt"
Oct 30 20:57:53.837178 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:53.837064    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62a6adf7-d17a-4fec-a707-a2d549b67753-tls-certs\") pod \"simulator-w2-32a969-kserve-6cc8dfc44c-dwhlt\" (UID: \"62a6adf7-d17a-4fec-a707-a2d549b67753\") " pod="e2e-models-e2e-worker-w2-32a969/simulator-w2-32a969-kserve-6cc8dfc44c-dwhlt"
Oct 30 20:57:53.837178 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:53.837105    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62a6adf7-d17a-4fec-a707-a2d549b67753-tmp-dir\") pod \"simulator-w2-32a969-kserve-6cc8dfc44c-dwhlt\" (UID: \"62a6adf7-d17a-4fec-a707-a2d549b67753\") " pod="e2e-models-e2e-worker-w2-32a969/simulator-w2-32a969-kserve-6cc8dfc44c-dwhlt"
Oct 30 20:57:53.837178 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:53.837126    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hk2j\" (UniqueName: \"kubernetes.io/projected/62a6adf7-d17a-4fec-a707-a2d549b67753-kube-api-access-2hk2j\") pod \"simulator-w2-32a969-kserve-6cc8dfc44c-dwhlt\" (UID: \"62a6adf7-d17a-4fec-a707-a2d549b67753\") " pod="e2e-models-e2e-worker-w2-32a969/simulator-w2-32a969-kserve-6cc8dfc44c-dwhlt"
Oct 30 20:57:53.938472 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:53.938366    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62a6adf7-d17a-4fec-a707-a2d549b67753-model-cache\") pod \"simulator-w2-32a969-kserve-6cc8dfc44c-dwhlt\" (UID: \"62a6adf7-d17a-4fec-a707-a2d549b67753\") " pod="e2e-models-e2e-worker-w2-32a969/simulator-w2-32a969-kserve-6cc8dfc44c-dwhlt"
Oct 30 20:57:53.938472 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:53.938420    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62a6adf7-d17a-4fec-a707-a2d549b67753-tls-certs\") pod \"simulator-w2-32a969-kserve-6cc8dfc44c-dwhlt\" (UID: \"62a6adf7-d17a-4fec-a707-a2d549b67753\") " pod="e2e-models-e2e-worker-w2-32a969/simulator-w2-32a969-kserve-6cc8dfc44c-dwhlt"
Oct 30 20:57:53.938732 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:53.938526    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62a6adf7-d17a-4fec-a707-a2d549b67753-tmp-dir\") pod \"simulator-w2-32a969-kserve-6cc8dfc44c-dwhlt\" (UID: \"62a6adf7-d17a-4fec-a707-a2d549b67753\") " pod="e2e-models-e2e-worker-w2-32a969/simulator-w2-32a969-kserve-6cc8dfc44c-dwhlt"
Oct 30 20:57:53.938732 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:53.938559    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hk2j\" (UniqueName: \"kubernetes.io/projected/62a6adf7-d17a-4fec-a707-a2d549b67753-kube-api-access-2hk2j\") pod \"simulator-w2-32a969-kserve-6cc8dfc44c-dwhlt\" (UID: \"62a6adf7-d17a-4fec-a707-a2d549b67753\") " pod="e2e-models-e2e-worker-w2-32a969/simulator-w2-32a969-kserve-6cc8dfc44c-dwhlt"
Oct 30 20:57:53.938732 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:53.938626    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62a6adf7-d17a-4fec-a707-a2d549b67753-dshm\") pod \"simulator-w2-32a969-kserve-6cc8dfc44c-dwhlt\" (UID: \"62a6adf7-d17a-4fec-a707-a2d549b67753\") " pod="e2e-models-e2e-worker-w2-32a969/simulator-w2-32a969-kserve-6cc8dfc44c-dwhlt"
Oct 30 20:57:53.938732 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:53.938658    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62a6adf7-d17a-4fec-a707-a2d549b67753-home\") pod \"simulator-w2-32a969-kserve-6cc8dfc44c-dwhlt\" (UID: \"62a6adf7-d17a-4fec-a707-a2d549b67753\") " pod="e2e-models-e2e-worker-w2-32a969/simulator-w2-32a969-kserve-6cc8dfc44c-dwhlt"
Oct 30 20:57:53.939029 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:53.938975    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62a6adf7-d17a-4fec-a707-a2d549b67753-model-cache\") pod \"simulator-w2-32a969-kserve-6cc8dfc44c-dwhlt\" (UID: \"62a6adf7-d17a-4fec-a707-a2d549b67753\") " pod="e2e-models-e2e-worker-w2-32a969/simulator-w2-32a969-kserve-6cc8dfc44c-dwhlt"
Oct 30 20:57:53.939146 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:53.939047    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62a6adf7-d17a-4fec-a707-a2d549b67753-home\") pod \"simulator-w2-32a969-kserve-6cc8dfc44c-dwhlt\" (UID: \"62a6adf7-d17a-4fec-a707-a2d549b67753\") " pod="e2e-models-e2e-worker-w2-32a969/simulator-w2-32a969-kserve-6cc8dfc44c-dwhlt"
Oct 30 20:57:53.939356 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:53.939309    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62a6adf7-d17a-4fec-a707-a2d549b67753-tmp-dir\") pod \"simulator-w2-32a969-kserve-6cc8dfc44c-dwhlt\" (UID: \"62a6adf7-d17a-4fec-a707-a2d549b67753\") " pod="e2e-models-e2e-worker-w2-32a969/simulator-w2-32a969-kserve-6cc8dfc44c-dwhlt"
Oct 30 20:57:53.944514 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:53.944490    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62a6adf7-d17a-4fec-a707-a2d549b67753-tls-certs\") pod \"simulator-w2-32a969-kserve-6cc8dfc44c-dwhlt\" (UID: \"62a6adf7-d17a-4fec-a707-a2d549b67753\") " pod="e2e-models-e2e-worker-w2-32a969/simulator-w2-32a969-kserve-6cc8dfc44c-dwhlt"
Oct 30 20:57:53.946956 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:53.946934    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62a6adf7-d17a-4fec-a707-a2d549b67753-dshm\") pod \"simulator-w2-32a969-kserve-6cc8dfc44c-dwhlt\" (UID: \"62a6adf7-d17a-4fec-a707-a2d549b67753\") " pod="e2e-models-e2e-worker-w2-32a969/simulator-w2-32a969-kserve-6cc8dfc44c-dwhlt"
Oct 30 20:57:53.954041 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:53.954008    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hk2j\" (UniqueName: \"kubernetes.io/projected/62a6adf7-d17a-4fec-a707-a2d549b67753-kube-api-access-2hk2j\") pod \"simulator-w2-32a969-kserve-6cc8dfc44c-dwhlt\" (UID: \"62a6adf7-d17a-4fec-a707-a2d549b67753\") " pod="e2e-models-e2e-worker-w2-32a969/simulator-w2-32a969-kserve-6cc8dfc44c-dwhlt"
Oct 30 20:57:54.044615 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:54.044577    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-32a969/simulator-w2-32a969-kserve-6cc8dfc44c-dwhlt"
Oct 30 20:57:54.332990 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:54.332958    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-32a969/simulator-w2-32a969-kserve-6cc8dfc44c-dwhlt"]
Oct 30 20:57:54.334517 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:57:54.334487    4521 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod62a6adf7_d17a_4fec_a707_a2d549b67753.slice/crio-ef10ad04f689458a8bcd631fda475ad89f00e222b8ec93f2ffbf197f2aafebf0 WatchSource:0}: Error finding container ef10ad04f689458a8bcd631fda475ad89f00e222b8ec93f2ffbf197f2aafebf0: Status 404 returned error can't find the container with id ef10ad04f689458a8bcd631fda475ad89f00e222b8ec93f2ffbf197f2aafebf0
Oct 30 20:57:54.738859 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:54.738810    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-32a969/simulator-w2-32a969-kserve-6cc8dfc44c-dwhlt" event={"ID":"62a6adf7-d17a-4fec-a707-a2d549b67753","Type":"ContainerStarted","Data":"ef10ad04f689458a8bcd631fda475ad89f00e222b8ec93f2ffbf197f2aafebf0"}
Oct 30 20:57:56.311931 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:56.311864    4521 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-5a8f94/simulator-w3-5a8f94-kserve-77cc88cc89-n4sl6"]
Oct 30 20:57:56.316067 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:56.316038    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-5a8f94/simulator-w3-5a8f94-kserve-77cc88cc89-n4sl6"
Oct 30 20:57:56.327038 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:56.327006    4521 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-5a8f94\"/\"simulator-w3-5a8f94-kserve-self-signed-certs\""
Oct 30 20:57:56.331821 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:56.331793    4521 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-5a8f94\"/\"default-dockercfg-fdmqv\""
Oct 30 20:57:56.333267 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:56.333243    4521 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-5a8f94\"/\"kube-root-ca.crt\""
Oct 30 20:57:56.333401 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:56.333367    4521 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-5a8f94\"/\"openshift-service-ca.crt\""
Oct 30 20:57:56.335023 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:56.335002    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-5a8f94/simulator-w3-5a8f94-kserve-77cc88cc89-n4sl6"]
Oct 30 20:57:56.479259 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:56.479222    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1fa6900c-eee2-4f00-b0ec-248def48dfc4-dshm\") pod \"simulator-w3-5a8f94-kserve-77cc88cc89-n4sl6\" (UID: \"1fa6900c-eee2-4f00-b0ec-248def48dfc4\") " pod="e2e-models-e2e-worker-w3-5a8f94/simulator-w3-5a8f94-kserve-77cc88cc89-n4sl6"
Oct 30 20:57:56.479460 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:56.479273    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fa6900c-eee2-4f00-b0ec-248def48dfc4-tmp-dir\") pod \"simulator-w3-5a8f94-kserve-77cc88cc89-n4sl6\" (UID: \"1fa6900c-eee2-4f00-b0ec-248def48dfc4\") " pod="e2e-models-e2e-worker-w3-5a8f94/simulator-w3-5a8f94-kserve-77cc88cc89-n4sl6"
Oct 30 20:57:56.479460 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:56.479434    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xljl7\" (UniqueName: \"kubernetes.io/projected/1fa6900c-eee2-4f00-b0ec-248def48dfc4-kube-api-access-xljl7\") pod \"simulator-w3-5a8f94-kserve-77cc88cc89-n4sl6\" (UID: \"1fa6900c-eee2-4f00-b0ec-248def48dfc4\") " pod="e2e-models-e2e-worker-w3-5a8f94/simulator-w3-5a8f94-kserve-77cc88cc89-n4sl6"
Oct 30 20:57:56.479598 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:56.479548    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1fa6900c-eee2-4f00-b0ec-248def48dfc4-model-cache\") pod \"simulator-w3-5a8f94-kserve-77cc88cc89-n4sl6\" (UID: \"1fa6900c-eee2-4f00-b0ec-248def48dfc4\") " pod="e2e-models-e2e-worker-w3-5a8f94/simulator-w3-5a8f94-kserve-77cc88cc89-n4sl6"
Oct 30 20:57:56.479672 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:56.479644    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1fa6900c-eee2-4f00-b0ec-248def48dfc4-home\") pod \"simulator-w3-5a8f94-kserve-77cc88cc89-n4sl6\" (UID: \"1fa6900c-eee2-4f00-b0ec-248def48dfc4\") " pod="e2e-models-e2e-worker-w3-5a8f94/simulator-w3-5a8f94-kserve-77cc88cc89-n4sl6"
Oct 30 20:57:56.479772 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:56.479680    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1fa6900c-eee2-4f00-b0ec-248def48dfc4-tls-certs\") pod \"simulator-w3-5a8f94-kserve-77cc88cc89-n4sl6\" (UID: \"1fa6900c-eee2-4f00-b0ec-248def48dfc4\") " pod="e2e-models-e2e-worker-w3-5a8f94/simulator-w3-5a8f94-kserve-77cc88cc89-n4sl6"
Oct 30 20:57:56.580670 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:56.580582    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xljl7\" (UniqueName: \"kubernetes.io/projected/1fa6900c-eee2-4f00-b0ec-248def48dfc4-kube-api-access-xljl7\") pod \"simulator-w3-5a8f94-kserve-77cc88cc89-n4sl6\" (UID: \"1fa6900c-eee2-4f00-b0ec-248def48dfc4\") " pod="e2e-models-e2e-worker-w3-5a8f94/simulator-w3-5a8f94-kserve-77cc88cc89-n4sl6"
Oct 30 20:57:56.580670 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:56.580632    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1fa6900c-eee2-4f00-b0ec-248def48dfc4-model-cache\") pod \"simulator-w3-5a8f94-kserve-77cc88cc89-n4sl6\" (UID: \"1fa6900c-eee2-4f00-b0ec-248def48dfc4\") " pod="e2e-models-e2e-worker-w3-5a8f94/simulator-w3-5a8f94-kserve-77cc88cc89-n4sl6"
Oct 30 20:57:56.580894 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:56.580813    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1fa6900c-eee2-4f00-b0ec-248def48dfc4-home\") pod \"simulator-w3-5a8f94-kserve-77cc88cc89-n4sl6\" (UID: \"1fa6900c-eee2-4f00-b0ec-248def48dfc4\") " pod="e2e-models-e2e-worker-w3-5a8f94/simulator-w3-5a8f94-kserve-77cc88cc89-n4sl6"
Oct 30 20:57:56.580894 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:56.580855    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1fa6900c-eee2-4f00-b0ec-248def48dfc4-tls-certs\") pod \"simulator-w3-5a8f94-kserve-77cc88cc89-n4sl6\" (UID: \"1fa6900c-eee2-4f00-b0ec-248def48dfc4\") " pod="e2e-models-e2e-worker-w3-5a8f94/simulator-w3-5a8f94-kserve-77cc88cc89-n4sl6"
Oct 30 20:57:56.580993 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:56.580933    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1fa6900c-eee2-4f00-b0ec-248def48dfc4-dshm\") pod \"simulator-w3-5a8f94-kserve-77cc88cc89-n4sl6\" (UID: \"1fa6900c-eee2-4f00-b0ec-248def48dfc4\") " pod="e2e-models-e2e-worker-w3-5a8f94/simulator-w3-5a8f94-kserve-77cc88cc89-n4sl6"
Oct 30 20:57:56.580993 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:56.580963    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fa6900c-eee2-4f00-b0ec-248def48dfc4-tmp-dir\") pod \"simulator-w3-5a8f94-kserve-77cc88cc89-n4sl6\" (UID: \"1fa6900c-eee2-4f00-b0ec-248def48dfc4\") " pod="e2e-models-e2e-worker-w3-5a8f94/simulator-w3-5a8f94-kserve-77cc88cc89-n4sl6"
Oct 30 20:57:56.581098 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:56.581068    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1fa6900c-eee2-4f00-b0ec-248def48dfc4-model-cache\") pod \"simulator-w3-5a8f94-kserve-77cc88cc89-n4sl6\" (UID: \"1fa6900c-eee2-4f00-b0ec-248def48dfc4\") " pod="e2e-models-e2e-worker-w3-5a8f94/simulator-w3-5a8f94-kserve-77cc88cc89-n4sl6"
Oct 30 20:57:56.581149 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:56.581127    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1fa6900c-eee2-4f00-b0ec-248def48dfc4-home\") pod \"simulator-w3-5a8f94-kserve-77cc88cc89-n4sl6\" (UID: \"1fa6900c-eee2-4f00-b0ec-248def48dfc4\") " pod="e2e-models-e2e-worker-w3-5a8f94/simulator-w3-5a8f94-kserve-77cc88cc89-n4sl6"
Oct 30 20:57:56.581397 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:56.581347    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fa6900c-eee2-4f00-b0ec-248def48dfc4-tmp-dir\") pod \"simulator-w3-5a8f94-kserve-77cc88cc89-n4sl6\" (UID: \"1fa6900c-eee2-4f00-b0ec-248def48dfc4\") " pod="e2e-models-e2e-worker-w3-5a8f94/simulator-w3-5a8f94-kserve-77cc88cc89-n4sl6"
Oct 30 20:57:56.583582 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:56.583553    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1fa6900c-eee2-4f00-b0ec-248def48dfc4-dshm\") pod \"simulator-w3-5a8f94-kserve-77cc88cc89-n4sl6\" (UID: \"1fa6900c-eee2-4f00-b0ec-248def48dfc4\") " pod="e2e-models-e2e-worker-w3-5a8f94/simulator-w3-5a8f94-kserve-77cc88cc89-n4sl6"
Oct 30 20:57:56.583710 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:56.583656    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1fa6900c-eee2-4f00-b0ec-248def48dfc4-tls-certs\") pod \"simulator-w3-5a8f94-kserve-77cc88cc89-n4sl6\" (UID: \"1fa6900c-eee2-4f00-b0ec-248def48dfc4\") " pod="e2e-models-e2e-worker-w3-5a8f94/simulator-w3-5a8f94-kserve-77cc88cc89-n4sl6"
Oct 30 20:57:56.600006 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:56.599972    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xljl7\" (UniqueName: \"kubernetes.io/projected/1fa6900c-eee2-4f00-b0ec-248def48dfc4-kube-api-access-xljl7\") pod \"simulator-w3-5a8f94-kserve-77cc88cc89-n4sl6\" (UID: \"1fa6900c-eee2-4f00-b0ec-248def48dfc4\") " pod="e2e-models-e2e-worker-w3-5a8f94/simulator-w3-5a8f94-kserve-77cc88cc89-n4sl6"
Oct 30 20:57:56.634079 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:56.634029    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-5a8f94/simulator-w3-5a8f94-kserve-77cc88cc89-n4sl6"
Oct 30 20:57:56.755042 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:56.755006    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-32a969/simulator-w2-32a969-kserve-6cc8dfc44c-dwhlt" event={"ID":"62a6adf7-d17a-4fec-a707-a2d549b67753","Type":"ContainerStarted","Data":"d0cc9665eaaba2041a8be6760b20929fefb206724c54b74e9da703a83acecc51"}
Oct 30 20:57:56.800838 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:56.800776    4521 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-32a969/simulator-w2-32a969-kserve-6cc8dfc44c-dwhlt" podStartSLOduration=2.375290646 podStartE2EDuration="3.800758733s" totalImagesPullingTime="1.425468087s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:57:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:57:56.798412204 +0000 UTC m=+1144.144033466" watchObservedRunningTime="2026-10-30 20:57:56.800758733 +0000 UTC m=+1144.146380001"
Oct 30 20:57:56.921121 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:56.921079    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-5a8f94/simulator-w3-5a8f94-kserve-77cc88cc89-n4sl6"]
Oct 30 20:57:56.921703 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:57:56.921678    4521 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1fa6900c_eee2_4f00_b0ec_248def48dfc4.slice/crio-511ae6fb670346702d3657af83f03c55f096748f636134b00370d78cff90ee85 WatchSource:0}: Error finding container 511ae6fb670346702d3657af83f03c55f096748f636134b00370d78cff90ee85: Status 404 returned error can't find the container with id 511ae6fb670346702d3657af83f03c55f096748f636134b00370d78cff90ee85
Oct 30 20:57:57.045528 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:57.045494    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-32a969/simulator-w2-32a969-kserve-6cc8dfc44c-dwhlt"
Oct 30 20:57:57.764228 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:57.764191    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-5a8f94/simulator-w3-5a8f94-kserve-77cc88cc89-n4sl6" event={"ID":"1fa6900c-eee2-4f00-b0ec-248def48dfc4","Type":"ContainerStarted","Data":"155ef94dbc2d28a469b16f4b47f77b5b424a93c6c7b687308f435f40ee44d4cc"}
Oct 30 20:57:57.764760 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:57.764249    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-5a8f94/simulator-w3-5a8f94-kserve-77cc88cc89-n4sl6" event={"ID":"1fa6900c-eee2-4f00-b0ec-248def48dfc4","Type":"ContainerStarted","Data":"511ae6fb670346702d3657af83f03c55f096748f636134b00370d78cff90ee85"}
Oct 30 20:57:57.810478 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:57.810409    4521 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-5a8f94/simulator-w3-5a8f94-kserve-77cc88cc89-n4sl6" podStartSLOduration=1.810391439 podStartE2EDuration="1.810391439s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:57:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:57:57.809016 +0000 UTC m=+1145.154637238" watchObservedRunningTime="2026-10-30 20:57:57.810391439 +0000 UTC m=+1145.156012679"
Oct 30 20:57:58.635344 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:57:58.635289    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5a8f94/simulator-w3-5a8f94-kserve-77cc88cc89-n4sl6"
Oct 30 20:58:02.494851 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:02.494807    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4kxb"]
Oct 30 20:58:04.044981 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:04.044942    4521 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-32a969/simulator-w2-32a969-kserve-6cc8dfc44c-dwhlt"
Oct 30 20:58:04.058220 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:04.058189    4521 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-32a969/simulator-w2-32a969-kserve-6cc8dfc44c-dwhlt"
Oct 30 20:58:04.821250 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:04.821218    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-32a969/simulator-w2-32a969-kserve-6cc8dfc44c-dwhlt"
Oct 30 20:58:06.635202 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:06.635164    4521 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-5a8f94/simulator-w3-5a8f94-kserve-77cc88cc89-n4sl6"
Oct 30 20:58:06.648468 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:06.648434    4521 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-5a8f94/simulator-w3-5a8f94-kserve-77cc88cc89-n4sl6"
Oct 30 20:58:06.826558 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:06.826531    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5a8f94/simulator-w3-5a8f94-kserve-77cc88cc89-n4sl6"
Oct 30 20:58:11.781190 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:11.781147    4521 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-4a0c5b/premium-w0-4a0c5b-kserve-85dcf89f57-mscdn"]
Oct 30 20:58:11.785303 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:11.785277    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-4a0c5b/premium-w0-4a0c5b-kserve-85dcf89f57-mscdn"
Oct 30 20:58:11.793903 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:11.793878    4521 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-4a0c5b\"/\"kube-root-ca.crt\""
Oct 30 20:58:11.795185 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:11.794931    4521 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-4a0c5b\"/\"default-dockercfg-46tcm\""
Oct 30 20:58:11.795185 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:11.795072    4521 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-4a0c5b\"/\"openshift-service-ca.crt\""
Oct 30 20:58:11.795185 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:11.795150    4521 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-4a0c5b\"/\"premium-w0-4a0c5b-kserve-self-signed-certs\""
Oct 30 20:58:11.798081 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:11.798056    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-4a0c5b/premium-w0-4a0c5b-kserve-85dcf89f57-mscdn"]
Oct 30 20:58:11.823593 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:11.823553    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00e42758-77ce-4663-adfa-2202c0aae24a-tls-certs\") pod \"premium-w0-4a0c5b-kserve-85dcf89f57-mscdn\" (UID: \"00e42758-77ce-4663-adfa-2202c0aae24a\") " pod="e2e-models-e2e-worker-w0-4a0c5b/premium-w0-4a0c5b-kserve-85dcf89f57-mscdn"
Oct 30 20:58:11.823771 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:11.823713    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00e42758-77ce-4663-adfa-2202c0aae24a-tmp-dir\") pod \"premium-w0-4a0c5b-kserve-85dcf89f57-mscdn\" (UID: \"00e42758-77ce-4663-adfa-2202c0aae24a\") " pod="e2e-models-e2e-worker-w0-4a0c5b/premium-w0-4a0c5b-kserve-85dcf89f57-mscdn"
Oct 30 20:58:11.823771 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:11.823741    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00e42758-77ce-4663-adfa-2202c0aae24a-dshm\") pod \"premium-w0-4a0c5b-kserve-85dcf89f57-mscdn\" (UID: \"00e42758-77ce-4663-adfa-2202c0aae24a\") " pod="e2e-models-e2e-worker-w0-4a0c5b/premium-w0-4a0c5b-kserve-85dcf89f57-mscdn"
Oct 30 20:58:11.823908 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:11.823812    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00e42758-77ce-4663-adfa-2202c0aae24a-home\") pod \"premium-w0-4a0c5b-kserve-85dcf89f57-mscdn\" (UID: \"00e42758-77ce-4663-adfa-2202c0aae24a\") " pod="e2e-models-e2e-worker-w0-4a0c5b/premium-w0-4a0c5b-kserve-85dcf89f57-mscdn"
Oct 30 20:58:11.823908 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:11.823843    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00e42758-77ce-4663-adfa-2202c0aae24a-model-cache\") pod \"premium-w0-4a0c5b-kserve-85dcf89f57-mscdn\" (UID: \"00e42758-77ce-4663-adfa-2202c0aae24a\") " pod="e2e-models-e2e-worker-w0-4a0c5b/premium-w0-4a0c5b-kserve-85dcf89f57-mscdn"
Oct 30 20:58:11.823908 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:11.823880    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z942d\" (UniqueName: \"kubernetes.io/projected/00e42758-77ce-4663-adfa-2202c0aae24a-kube-api-access-z942d\") pod \"premium-w0-4a0c5b-kserve-85dcf89f57-mscdn\" (UID: \"00e42758-77ce-4663-adfa-2202c0aae24a\") " pod="e2e-models-e2e-worker-w0-4a0c5b/premium-w0-4a0c5b-kserve-85dcf89f57-mscdn"
Oct 30 20:58:11.924836 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:11.924797    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00e42758-77ce-4663-adfa-2202c0aae24a-tmp-dir\") pod \"premium-w0-4a0c5b-kserve-85dcf89f57-mscdn\" (UID: \"00e42758-77ce-4663-adfa-2202c0aae24a\") " pod="e2e-models-e2e-worker-w0-4a0c5b/premium-w0-4a0c5b-kserve-85dcf89f57-mscdn"
Oct 30 20:58:11.925062 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:11.924842    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00e42758-77ce-4663-adfa-2202c0aae24a-dshm\") pod \"premium-w0-4a0c5b-kserve-85dcf89f57-mscdn\" (UID: \"00e42758-77ce-4663-adfa-2202c0aae24a\") " pod="e2e-models-e2e-worker-w0-4a0c5b/premium-w0-4a0c5b-kserve-85dcf89f57-mscdn"
Oct 30 20:58:11.925062 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:11.924908    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00e42758-77ce-4663-adfa-2202c0aae24a-home\") pod \"premium-w0-4a0c5b-kserve-85dcf89f57-mscdn\" (UID: \"00e42758-77ce-4663-adfa-2202c0aae24a\") " pod="e2e-models-e2e-worker-w0-4a0c5b/premium-w0-4a0c5b-kserve-85dcf89f57-mscdn"
Oct 30 20:58:11.925062 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:11.924941    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00e42758-77ce-4663-adfa-2202c0aae24a-model-cache\") pod \"premium-w0-4a0c5b-kserve-85dcf89f57-mscdn\" (UID: \"00e42758-77ce-4663-adfa-2202c0aae24a\") " pod="e2e-models-e2e-worker-w0-4a0c5b/premium-w0-4a0c5b-kserve-85dcf89f57-mscdn"
Oct 30 20:58:11.925062 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:11.924976    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z942d\" (UniqueName: \"kubernetes.io/projected/00e42758-77ce-4663-adfa-2202c0aae24a-kube-api-access-z942d\") pod \"premium-w0-4a0c5b-kserve-85dcf89f57-mscdn\" (UID: \"00e42758-77ce-4663-adfa-2202c0aae24a\") " pod="e2e-models-e2e-worker-w0-4a0c5b/premium-w0-4a0c5b-kserve-85dcf89f57-mscdn"
Oct 30 20:58:11.925062 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:11.925058    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00e42758-77ce-4663-adfa-2202c0aae24a-tls-certs\") pod \"premium-w0-4a0c5b-kserve-85dcf89f57-mscdn\" (UID: \"00e42758-77ce-4663-adfa-2202c0aae24a\") " pod="e2e-models-e2e-worker-w0-4a0c5b/premium-w0-4a0c5b-kserve-85dcf89f57-mscdn"
Oct 30 20:58:11.925658 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:11.925627    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00e42758-77ce-4663-adfa-2202c0aae24a-tmp-dir\") pod \"premium-w0-4a0c5b-kserve-85dcf89f57-mscdn\" (UID: \"00e42758-77ce-4663-adfa-2202c0aae24a\") " pod="e2e-models-e2e-worker-w0-4a0c5b/premium-w0-4a0c5b-kserve-85dcf89f57-mscdn"
Oct 30 20:58:11.925786 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:11.925712    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00e42758-77ce-4663-adfa-2202c0aae24a-home\") pod \"premium-w0-4a0c5b-kserve-85dcf89f57-mscdn\" (UID: \"00e42758-77ce-4663-adfa-2202c0aae24a\") " pod="e2e-models-e2e-worker-w0-4a0c5b/premium-w0-4a0c5b-kserve-85dcf89f57-mscdn"
Oct 30 20:58:11.925929 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:11.925904    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00e42758-77ce-4663-adfa-2202c0aae24a-model-cache\") pod \"premium-w0-4a0c5b-kserve-85dcf89f57-mscdn\" (UID: \"00e42758-77ce-4663-adfa-2202c0aae24a\") " pod="e2e-models-e2e-worker-w0-4a0c5b/premium-w0-4a0c5b-kserve-85dcf89f57-mscdn"
Oct 30 20:58:11.927972 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:11.927942    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00e42758-77ce-4663-adfa-2202c0aae24a-dshm\") pod \"premium-w0-4a0c5b-kserve-85dcf89f57-mscdn\" (UID: \"00e42758-77ce-4663-adfa-2202c0aae24a\") " pod="e2e-models-e2e-worker-w0-4a0c5b/premium-w0-4a0c5b-kserve-85dcf89f57-mscdn"
Oct 30 20:58:11.928511 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:11.928491    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00e42758-77ce-4663-adfa-2202c0aae24a-tls-certs\") pod \"premium-w0-4a0c5b-kserve-85dcf89f57-mscdn\" (UID: \"00e42758-77ce-4663-adfa-2202c0aae24a\") " pod="e2e-models-e2e-worker-w0-4a0c5b/premium-w0-4a0c5b-kserve-85dcf89f57-mscdn"
Oct 30 20:58:11.943206 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:11.943138    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z942d\" (UniqueName: \"kubernetes.io/projected/00e42758-77ce-4663-adfa-2202c0aae24a-kube-api-access-z942d\") pod \"premium-w0-4a0c5b-kserve-85dcf89f57-mscdn\" (UID: \"00e42758-77ce-4663-adfa-2202c0aae24a\") " pod="e2e-models-e2e-worker-w0-4a0c5b/premium-w0-4a0c5b-kserve-85dcf89f57-mscdn"
Oct 30 20:58:12.099870 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:12.099787    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-4a0c5b/premium-w0-4a0c5b-kserve-85dcf89f57-mscdn"
Oct 30 20:58:12.387564 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:12.387536    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-4a0c5b/premium-w0-4a0c5b-kserve-85dcf89f57-mscdn"]
Oct 30 20:58:12.389358 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:58:12.389298    4521 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod00e42758_77ce_4663_adfa_2202c0aae24a.slice/crio-bec9b5108a5763f427eafb09de63f6ae882559cbc1f2006e8e6fa73ec57f479f WatchSource:0}: Error finding container bec9b5108a5763f427eafb09de63f6ae882559cbc1f2006e8e6fa73ec57f479f: Status 404 returned error can't find the container with id bec9b5108a5763f427eafb09de63f6ae882559cbc1f2006e8e6fa73ec57f479f
Oct 30 20:58:12.422247 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:12.422213    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4kxb"]
Oct 30 20:58:12.855915 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:12.855874    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-4a0c5b/premium-w0-4a0c5b-kserve-85dcf89f57-mscdn" event={"ID":"00e42758-77ce-4663-adfa-2202c0aae24a","Type":"ContainerStarted","Data":"ecc44c3983a6afc4815071bbc2e79af31a0e055659cee676c6271a9969018ae1"}
Oct 30 20:58:12.856355 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:12.855933    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-4a0c5b/premium-w0-4a0c5b-kserve-85dcf89f57-mscdn" event={"ID":"00e42758-77ce-4663-adfa-2202c0aae24a","Type":"ContainerStarted","Data":"bec9b5108a5763f427eafb09de63f6ae882559cbc1f2006e8e6fa73ec57f479f"}
Oct 30 20:58:12.878425 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:12.878368    4521 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-4a0c5b/premium-w0-4a0c5b-kserve-85dcf89f57-mscdn" podStartSLOduration=1.8783040450000001 podStartE2EDuration="1.878304045s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:58:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:58:12.876769497 +0000 UTC m=+1160.222390735" watchObservedRunningTime="2026-10-30 20:58:12.878304045 +0000 UTC m=+1160.223925283"
Oct 30 20:58:13.100917 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:13.100877    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4a0c5b/premium-w0-4a0c5b-kserve-85dcf89f57-mscdn"
Oct 30 20:58:16.433317 ip-10-0-131-125 kubenswrapper[4521]: E1030 20:58:16.433270    4521 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lkn2k" containerName="extractor"
Oct 30 20:58:20.732089 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:20.732047    4521 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-5a8f94/premium-w3-5a8f94-kserve-bb96b6848-5hk8h"]
Oct 30 20:58:20.741156 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:20.741123    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-5a8f94/premium-w3-5a8f94-kserve-bb96b6848-5hk8h"
Oct 30 20:58:20.747968 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:20.747934    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-5a8f94/premium-w3-5a8f94-kserve-bb96b6848-5hk8h"]
Oct 30 20:58:20.750695 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:20.750660    4521 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-5a8f94\"/\"premium-w3-5a8f94-kserve-self-signed-certs\""
Oct 30 20:58:20.824821 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:20.824776    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24ccc37d-027a-4097-8da5-04df417aef94-tmp-dir\") pod \"premium-w3-5a8f94-kserve-bb96b6848-5hk8h\" (UID: \"24ccc37d-027a-4097-8da5-04df417aef94\") " pod="e2e-models-e2e-worker-w3-5a8f94/premium-w3-5a8f94-kserve-bb96b6848-5hk8h"
Oct 30 20:58:20.825016 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:20.824866    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24ccc37d-027a-4097-8da5-04df417aef94-dshm\") pod \"premium-w3-5a8f94-kserve-bb96b6848-5hk8h\" (UID: \"24ccc37d-027a-4097-8da5-04df417aef94\") " pod="e2e-models-e2e-worker-w3-5a8f94/premium-w3-5a8f94-kserve-bb96b6848-5hk8h"
Oct 30 20:58:20.825016 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:20.824900    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24ccc37d-027a-4097-8da5-04df417aef94-home\") pod \"premium-w3-5a8f94-kserve-bb96b6848-5hk8h\" (UID: \"24ccc37d-027a-4097-8da5-04df417aef94\") " pod="e2e-models-e2e-worker-w3-5a8f94/premium-w3-5a8f94-kserve-bb96b6848-5hk8h"
Oct 30 20:58:20.825016 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:20.824934    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkrx6\" (UniqueName: \"kubernetes.io/projected/24ccc37d-027a-4097-8da5-04df417aef94-kube-api-access-dkrx6\") pod \"premium-w3-5a8f94-kserve-bb96b6848-5hk8h\" (UID: \"24ccc37d-027a-4097-8da5-04df417aef94\") " pod="e2e-models-e2e-worker-w3-5a8f94/premium-w3-5a8f94-kserve-bb96b6848-5hk8h"
Oct 30 20:58:20.825016 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:20.824975    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24ccc37d-027a-4097-8da5-04df417aef94-model-cache\") pod \"premium-w3-5a8f94-kserve-bb96b6848-5hk8h\" (UID: \"24ccc37d-027a-4097-8da5-04df417aef94\") " pod="e2e-models-e2e-worker-w3-5a8f94/premium-w3-5a8f94-kserve-bb96b6848-5hk8h"
Oct 30 20:58:20.825016 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:20.825010    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24ccc37d-027a-4097-8da5-04df417aef94-tls-certs\") pod \"premium-w3-5a8f94-kserve-bb96b6848-5hk8h\" (UID: \"24ccc37d-027a-4097-8da5-04df417aef94\") " pod="e2e-models-e2e-worker-w3-5a8f94/premium-w3-5a8f94-kserve-bb96b6848-5hk8h"
Oct 30 20:58:20.925704 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:20.925671    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24ccc37d-027a-4097-8da5-04df417aef94-dshm\") pod \"premium-w3-5a8f94-kserve-bb96b6848-5hk8h\" (UID: \"24ccc37d-027a-4097-8da5-04df417aef94\") " pod="e2e-models-e2e-worker-w3-5a8f94/premium-w3-5a8f94-kserve-bb96b6848-5hk8h"
Oct 30 20:58:20.925916 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:20.925721    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24ccc37d-027a-4097-8da5-04df417aef94-home\") pod \"premium-w3-5a8f94-kserve-bb96b6848-5hk8h\" (UID: \"24ccc37d-027a-4097-8da5-04df417aef94\") " pod="e2e-models-e2e-worker-w3-5a8f94/premium-w3-5a8f94-kserve-bb96b6848-5hk8h"
Oct 30 20:58:20.925916 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:20.925757    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkrx6\" (UniqueName: \"kubernetes.io/projected/24ccc37d-027a-4097-8da5-04df417aef94-kube-api-access-dkrx6\") pod \"premium-w3-5a8f94-kserve-bb96b6848-5hk8h\" (UID: \"24ccc37d-027a-4097-8da5-04df417aef94\") " pod="e2e-models-e2e-worker-w3-5a8f94/premium-w3-5a8f94-kserve-bb96b6848-5hk8h"
Oct 30 20:58:20.925916 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:20.925798    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24ccc37d-027a-4097-8da5-04df417aef94-model-cache\") pod \"premium-w3-5a8f94-kserve-bb96b6848-5hk8h\" (UID: \"24ccc37d-027a-4097-8da5-04df417aef94\") " pod="e2e-models-e2e-worker-w3-5a8f94/premium-w3-5a8f94-kserve-bb96b6848-5hk8h"
Oct 30 20:58:20.925916 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:20.925835    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24ccc37d-027a-4097-8da5-04df417aef94-tls-certs\") pod \"premium-w3-5a8f94-kserve-bb96b6848-5hk8h\" (UID: \"24ccc37d-027a-4097-8da5-04df417aef94\") " pod="e2e-models-e2e-worker-w3-5a8f94/premium-w3-5a8f94-kserve-bb96b6848-5hk8h"
Oct 30 20:58:20.925916 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:20.925891    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24ccc37d-027a-4097-8da5-04df417aef94-tmp-dir\") pod \"premium-w3-5a8f94-kserve-bb96b6848-5hk8h\" (UID: \"24ccc37d-027a-4097-8da5-04df417aef94\") " pod="e2e-models-e2e-worker-w3-5a8f94/premium-w3-5a8f94-kserve-bb96b6848-5hk8h"
Oct 30 20:58:20.926355 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:20.926317    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24ccc37d-027a-4097-8da5-04df417aef94-tmp-dir\") pod \"premium-w3-5a8f94-kserve-bb96b6848-5hk8h\" (UID: \"24ccc37d-027a-4097-8da5-04df417aef94\") " pod="e2e-models-e2e-worker-w3-5a8f94/premium-w3-5a8f94-kserve-bb96b6848-5hk8h"
Oct 30 20:58:20.926516 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:20.926490    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24ccc37d-027a-4097-8da5-04df417aef94-home\") pod \"premium-w3-5a8f94-kserve-bb96b6848-5hk8h\" (UID: \"24ccc37d-027a-4097-8da5-04df417aef94\") " pod="e2e-models-e2e-worker-w3-5a8f94/premium-w3-5a8f94-kserve-bb96b6848-5hk8h"
Oct 30 20:58:20.926516 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:20.926513    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24ccc37d-027a-4097-8da5-04df417aef94-model-cache\") pod \"premium-w3-5a8f94-kserve-bb96b6848-5hk8h\" (UID: \"24ccc37d-027a-4097-8da5-04df417aef94\") " pod="e2e-models-e2e-worker-w3-5a8f94/premium-w3-5a8f94-kserve-bb96b6848-5hk8h"
Oct 30 20:58:20.928594 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:20.928571    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24ccc37d-027a-4097-8da5-04df417aef94-dshm\") pod \"premium-w3-5a8f94-kserve-bb96b6848-5hk8h\" (UID: \"24ccc37d-027a-4097-8da5-04df417aef94\") " pod="e2e-models-e2e-worker-w3-5a8f94/premium-w3-5a8f94-kserve-bb96b6848-5hk8h"
Oct 30 20:58:20.928730 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:20.928712    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24ccc37d-027a-4097-8da5-04df417aef94-tls-certs\") pod \"premium-w3-5a8f94-kserve-bb96b6848-5hk8h\" (UID: \"24ccc37d-027a-4097-8da5-04df417aef94\") " pod="e2e-models-e2e-worker-w3-5a8f94/premium-w3-5a8f94-kserve-bb96b6848-5hk8h"
Oct 30 20:58:20.938572 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:20.938538    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkrx6\" (UniqueName: \"kubernetes.io/projected/24ccc37d-027a-4097-8da5-04df417aef94-kube-api-access-dkrx6\") pod \"premium-w3-5a8f94-kserve-bb96b6848-5hk8h\" (UID: \"24ccc37d-027a-4097-8da5-04df417aef94\") " pod="e2e-models-e2e-worker-w3-5a8f94/premium-w3-5a8f94-kserve-bb96b6848-5hk8h"
Oct 30 20:58:21.061485 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:21.061401    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-5a8f94/premium-w3-5a8f94-kserve-bb96b6848-5hk8h"
Oct 30 20:58:21.347429 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:21.347400    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-5a8f94/premium-w3-5a8f94-kserve-bb96b6848-5hk8h"]
Oct 30 20:58:21.349955 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:58:21.349928    4521 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod24ccc37d_027a_4097_8da5_04df417aef94.slice/crio-198a7db7117f283b4da4bece98ff16b4ab05bdd2e8ca9ed0399ce0186d7067b4 WatchSource:0}: Error finding container 198a7db7117f283b4da4bece98ff16b4ab05bdd2e8ca9ed0399ce0186d7067b4: Status 404 returned error can't find the container with id 198a7db7117f283b4da4bece98ff16b4ab05bdd2e8ca9ed0399ce0186d7067b4
Oct 30 20:58:21.909034 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:21.908992    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-5a8f94/premium-w3-5a8f94-kserve-bb96b6848-5hk8h" event={"ID":"24ccc37d-027a-4097-8da5-04df417aef94","Type":"ContainerStarted","Data":"47c33f2a9ce29670bd78c147b405d8d795a4c4b67ba92d3660b28bff3dd7026c"}
Oct 30 20:58:21.909581 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:21.909046    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-5a8f94/premium-w3-5a8f94-kserve-bb96b6848-5hk8h" event={"ID":"24ccc37d-027a-4097-8da5-04df417aef94","Type":"ContainerStarted","Data":"198a7db7117f283b4da4bece98ff16b4ab05bdd2e8ca9ed0399ce0186d7067b4"}
Oct 30 20:58:21.934399 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:21.934317    4521 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-5a8f94/premium-w3-5a8f94-kserve-bb96b6848-5hk8h" podStartSLOduration=1.934302354 podStartE2EDuration="1.934302354s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:58:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:58:21.931053716 +0000 UTC m=+1169.276674954" watchObservedRunningTime="2026-10-30 20:58:21.934302354 +0000 UTC m=+1169.279923635"
Oct 30 20:58:22.062382 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:22.062313    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5a8f94/premium-w3-5a8f94-kserve-bb96b6848-5hk8h"
Oct 30 20:58:22.100635 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:22.100595    4521 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-4a0c5b/premium-w0-4a0c5b-kserve-85dcf89f57-mscdn"
Oct 30 20:58:22.113876 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:22.113849    4521 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-4a0c5b/premium-w0-4a0c5b-kserve-85dcf89f57-mscdn"
Oct 30 20:58:22.927534 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:22.927504    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4a0c5b/premium-w0-4a0c5b-kserve-85dcf89f57-mscdn"
Oct 30 20:58:26.195833 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:26.195796    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4kxb"]
Oct 30 20:58:27.478003 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:27.477972    4521 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-65968b/premium-w5-65968b-kserve-67977485f7-c9jpv"]
Oct 30 20:58:27.484508 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:27.484467    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-65968b/premium-w5-65968b-kserve-67977485f7-c9jpv"
Oct 30 20:58:27.491964 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:27.491935    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-65968b/premium-w5-65968b-kserve-67977485f7-c9jpv"]
Oct 30 20:58:27.492266 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:27.492246    4521 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-65968b\"/\"kube-root-ca.crt\""
Oct 30 20:58:27.492620 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:27.492591    4521 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-65968b\"/\"default-dockercfg-xrc8s\""
Oct 30 20:58:27.494112 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:27.493871    4521 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-65968b\"/\"premium-w5-65968b-kserve-self-signed-certs\""
Oct 30 20:58:27.495545 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:27.495525    4521 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-65968b\"/\"openshift-service-ca.crt\""
Oct 30 20:58:27.593130 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:27.593087    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92b2dd65-8a83-436d-afa8-50e5f0d11fa0-home\") pod \"premium-w5-65968b-kserve-67977485f7-c9jpv\" (UID: \"92b2dd65-8a83-436d-afa8-50e5f0d11fa0\") " pod="e2e-models-e2e-worker-w5-65968b/premium-w5-65968b-kserve-67977485f7-c9jpv"
Oct 30 20:58:27.593130 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:27.593133    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92b2dd65-8a83-436d-afa8-50e5f0d11fa0-model-cache\") pod \"premium-w5-65968b-kserve-67977485f7-c9jpv\" (UID: \"92b2dd65-8a83-436d-afa8-50e5f0d11fa0\") " pod="e2e-models-e2e-worker-w5-65968b/premium-w5-65968b-kserve-67977485f7-c9jpv"
Oct 30 20:58:27.593421 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:27.593290    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92b2dd65-8a83-436d-afa8-50e5f0d11fa0-dshm\") pod \"premium-w5-65968b-kserve-67977485f7-c9jpv\" (UID: \"92b2dd65-8a83-436d-afa8-50e5f0d11fa0\") " pod="e2e-models-e2e-worker-w5-65968b/premium-w5-65968b-kserve-67977485f7-c9jpv"
Oct 30 20:58:27.593421 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:27.593408    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fw22s\" (UniqueName: \"kubernetes.io/projected/92b2dd65-8a83-436d-afa8-50e5f0d11fa0-kube-api-access-fw22s\") pod \"premium-w5-65968b-kserve-67977485f7-c9jpv\" (UID: \"92b2dd65-8a83-436d-afa8-50e5f0d11fa0\") " pod="e2e-models-e2e-worker-w5-65968b/premium-w5-65968b-kserve-67977485f7-c9jpv"
Oct 30 20:58:27.593547 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:27.593443    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92b2dd65-8a83-436d-afa8-50e5f0d11fa0-tmp-dir\") pod \"premium-w5-65968b-kserve-67977485f7-c9jpv\" (UID: \"92b2dd65-8a83-436d-afa8-50e5f0d11fa0\") " pod="e2e-models-e2e-worker-w5-65968b/premium-w5-65968b-kserve-67977485f7-c9jpv"
Oct 30 20:58:27.593547 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:27.593464    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92b2dd65-8a83-436d-afa8-50e5f0d11fa0-tls-certs\") pod \"premium-w5-65968b-kserve-67977485f7-c9jpv\" (UID: \"92b2dd65-8a83-436d-afa8-50e5f0d11fa0\") " pod="e2e-models-e2e-worker-w5-65968b/premium-w5-65968b-kserve-67977485f7-c9jpv"
Oct 30 20:58:27.693992 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:27.693955    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92b2dd65-8a83-436d-afa8-50e5f0d11fa0-dshm\") pod \"premium-w5-65968b-kserve-67977485f7-c9jpv\" (UID: \"92b2dd65-8a83-436d-afa8-50e5f0d11fa0\") " pod="e2e-models-e2e-worker-w5-65968b/premium-w5-65968b-kserve-67977485f7-c9jpv"
Oct 30 20:58:27.694204 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:27.694012    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fw22s\" (UniqueName: \"kubernetes.io/projected/92b2dd65-8a83-436d-afa8-50e5f0d11fa0-kube-api-access-fw22s\") pod \"premium-w5-65968b-kserve-67977485f7-c9jpv\" (UID: \"92b2dd65-8a83-436d-afa8-50e5f0d11fa0\") " pod="e2e-models-e2e-worker-w5-65968b/premium-w5-65968b-kserve-67977485f7-c9jpv"
Oct 30 20:58:27.694204 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:27.694050    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92b2dd65-8a83-436d-afa8-50e5f0d11fa0-tmp-dir\") pod \"premium-w5-65968b-kserve-67977485f7-c9jpv\" (UID: \"92b2dd65-8a83-436d-afa8-50e5f0d11fa0\") " pod="e2e-models-e2e-worker-w5-65968b/premium-w5-65968b-kserve-67977485f7-c9jpv"
Oct 30 20:58:27.694204 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:27.694082    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92b2dd65-8a83-436d-afa8-50e5f0d11fa0-tls-certs\") pod \"premium-w5-65968b-kserve-67977485f7-c9jpv\" (UID: \"92b2dd65-8a83-436d-afa8-50e5f0d11fa0\") " pod="e2e-models-e2e-worker-w5-65968b/premium-w5-65968b-kserve-67977485f7-c9jpv"
Oct 30 20:58:27.694512 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:27.694461    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92b2dd65-8a83-436d-afa8-50e5f0d11fa0-home\") pod \"premium-w5-65968b-kserve-67977485f7-c9jpv\" (UID: \"92b2dd65-8a83-436d-afa8-50e5f0d11fa0\") " pod="e2e-models-e2e-worker-w5-65968b/premium-w5-65968b-kserve-67977485f7-c9jpv"
Oct 30 20:58:27.695069 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:27.694733    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92b2dd65-8a83-436d-afa8-50e5f0d11fa0-model-cache\") pod \"premium-w5-65968b-kserve-67977485f7-c9jpv\" (UID: \"92b2dd65-8a83-436d-afa8-50e5f0d11fa0\") " pod="e2e-models-e2e-worker-w5-65968b/premium-w5-65968b-kserve-67977485f7-c9jpv"
Oct 30 20:58:27.695179 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:27.694531    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92b2dd65-8a83-436d-afa8-50e5f0d11fa0-tmp-dir\") pod \"premium-w5-65968b-kserve-67977485f7-c9jpv\" (UID: \"92b2dd65-8a83-436d-afa8-50e5f0d11fa0\") " pod="e2e-models-e2e-worker-w5-65968b/premium-w5-65968b-kserve-67977485f7-c9jpv"
Oct 30 20:58:27.695179 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:27.694783    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92b2dd65-8a83-436d-afa8-50e5f0d11fa0-home\") pod \"premium-w5-65968b-kserve-67977485f7-c9jpv\" (UID: \"92b2dd65-8a83-436d-afa8-50e5f0d11fa0\") " pod="e2e-models-e2e-worker-w5-65968b/premium-w5-65968b-kserve-67977485f7-c9jpv"
Oct 30 20:58:27.695179 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:27.695026    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92b2dd65-8a83-436d-afa8-50e5f0d11fa0-model-cache\") pod \"premium-w5-65968b-kserve-67977485f7-c9jpv\" (UID: \"92b2dd65-8a83-436d-afa8-50e5f0d11fa0\") " pod="e2e-models-e2e-worker-w5-65968b/premium-w5-65968b-kserve-67977485f7-c9jpv"
Oct 30 20:58:27.696829 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:27.696803    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92b2dd65-8a83-436d-afa8-50e5f0d11fa0-dshm\") pod \"premium-w5-65968b-kserve-67977485f7-c9jpv\" (UID: \"92b2dd65-8a83-436d-afa8-50e5f0d11fa0\") " pod="e2e-models-e2e-worker-w5-65968b/premium-w5-65968b-kserve-67977485f7-c9jpv"
Oct 30 20:58:27.696931 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:27.696848    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92b2dd65-8a83-436d-afa8-50e5f0d11fa0-tls-certs\") pod \"premium-w5-65968b-kserve-67977485f7-c9jpv\" (UID: \"92b2dd65-8a83-436d-afa8-50e5f0d11fa0\") " pod="e2e-models-e2e-worker-w5-65968b/premium-w5-65968b-kserve-67977485f7-c9jpv"
Oct 30 20:58:27.707187 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:27.707164    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fw22s\" (UniqueName: \"kubernetes.io/projected/92b2dd65-8a83-436d-afa8-50e5f0d11fa0-kube-api-access-fw22s\") pod \"premium-w5-65968b-kserve-67977485f7-c9jpv\" (UID: \"92b2dd65-8a83-436d-afa8-50e5f0d11fa0\") " pod="e2e-models-e2e-worker-w5-65968b/premium-w5-65968b-kserve-67977485f7-c9jpv"
Oct 30 20:58:27.809905 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:27.809820    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-65968b/premium-w5-65968b-kserve-67977485f7-c9jpv"
Oct 30 20:58:28.102588 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:28.102559    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-65968b/premium-w5-65968b-kserve-67977485f7-c9jpv"]
Oct 30 20:58:28.103463 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:58:28.103440    4521 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod92b2dd65_8a83_436d_afa8_50e5f0d11fa0.slice/crio-0a1316c5e82c70ca87620739b7f948403e37a79683bedce33aa4815201fdffc1 WatchSource:0}: Error finding container 0a1316c5e82c70ca87620739b7f948403e37a79683bedce33aa4815201fdffc1: Status 404 returned error can't find the container with id 0a1316c5e82c70ca87620739b7f948403e37a79683bedce33aa4815201fdffc1
Oct 30 20:58:28.958273 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:28.958220    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-65968b/premium-w5-65968b-kserve-67977485f7-c9jpv" event={"ID":"92b2dd65-8a83-436d-afa8-50e5f0d11fa0","Type":"ContainerStarted","Data":"f850ab3fba071258f1ebd3015d73e51d8c7a3c2027a8c8a5c70910ca96df1a37"}
Oct 30 20:58:28.958273 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:28.958275    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-65968b/premium-w5-65968b-kserve-67977485f7-c9jpv" event={"ID":"92b2dd65-8a83-436d-afa8-50e5f0d11fa0","Type":"ContainerStarted","Data":"0a1316c5e82c70ca87620739b7f948403e37a79683bedce33aa4815201fdffc1"}
Oct 30 20:58:28.978665 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:28.978605    4521 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-65968b/premium-w5-65968b-kserve-67977485f7-c9jpv" podStartSLOduration=1.978591325 podStartE2EDuration="1.978591325s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:58:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:58:28.978040224 +0000 UTC m=+1176.323661462" watchObservedRunningTime="2026-10-30 20:58:28.978591325 +0000 UTC m=+1176.324212563"
Oct 30 20:58:29.810938 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:29.810908    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-65968b/premium-w5-65968b-kserve-67977485f7-c9jpv"
Oct 30 20:58:30.987653 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:30.987610    4521 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-c4b379/premium-w1-c4b379-kserve-575f896d49-2n6gz"]
Oct 30 20:58:30.991729 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:30.991699    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-c4b379/premium-w1-c4b379-kserve-575f896d49-2n6gz"
Oct 30 20:58:30.999049 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:30.999011    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-c4b379/premium-w1-c4b379-kserve-575f896d49-2n6gz"]
Oct 30 20:58:30.999884 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:30.999863    4521 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-c4b379\"/\"openshift-service-ca.crt\""
Oct 30 20:58:31.000020 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:30.999936    4521 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-c4b379\"/\"premium-w1-c4b379-kserve-self-signed-certs\""
Oct 30 20:58:31.000020 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:31.000007    4521 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-c4b379\"/\"kube-root-ca.crt\""
Oct 30 20:58:31.000141 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:31.000091    4521 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-c4b379\"/\"default-dockercfg-b99dh\""
Oct 30 20:58:31.061636 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:31.061596    4521 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-5a8f94/premium-w3-5a8f94-kserve-bb96b6848-5hk8h"
Oct 30 20:58:31.074280 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:31.074247    4521 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-5a8f94/premium-w3-5a8f94-kserve-bb96b6848-5hk8h"
Oct 30 20:58:31.131399 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:31.131360    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2aac3e8e-d631-4b9c-bdbb-3b4b1df133a9-tls-certs\") pod \"premium-w1-c4b379-kserve-575f896d49-2n6gz\" (UID: \"2aac3e8e-d631-4b9c-bdbb-3b4b1df133a9\") " pod="e2e-models-e2e-worker-w1-c4b379/premium-w1-c4b379-kserve-575f896d49-2n6gz"
Oct 30 20:58:31.131694 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:31.131671    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2aac3e8e-d631-4b9c-bdbb-3b4b1df133a9-model-cache\") pod \"premium-w1-c4b379-kserve-575f896d49-2n6gz\" (UID: \"2aac3e8e-d631-4b9c-bdbb-3b4b1df133a9\") " pod="e2e-models-e2e-worker-w1-c4b379/premium-w1-c4b379-kserve-575f896d49-2n6gz"
Oct 30 20:58:31.131844 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:31.131826    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2aac3e8e-d631-4b9c-bdbb-3b4b1df133a9-dshm\") pod \"premium-w1-c4b379-kserve-575f896d49-2n6gz\" (UID: \"2aac3e8e-d631-4b9c-bdbb-3b4b1df133a9\") " pod="e2e-models-e2e-worker-w1-c4b379/premium-w1-c4b379-kserve-575f896d49-2n6gz"
Oct 30 20:58:31.131982 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:31.131967    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2aac3e8e-d631-4b9c-bdbb-3b4b1df133a9-tmp-dir\") pod \"premium-w1-c4b379-kserve-575f896d49-2n6gz\" (UID: \"2aac3e8e-d631-4b9c-bdbb-3b4b1df133a9\") " pod="e2e-models-e2e-worker-w1-c4b379/premium-w1-c4b379-kserve-575f896d49-2n6gz"
Oct 30 20:58:31.132244 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:31.132217    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lrj6\" (UniqueName: \"kubernetes.io/projected/2aac3e8e-d631-4b9c-bdbb-3b4b1df133a9-kube-api-access-7lrj6\") pod \"premium-w1-c4b379-kserve-575f896d49-2n6gz\" (UID: \"2aac3e8e-d631-4b9c-bdbb-3b4b1df133a9\") " pod="e2e-models-e2e-worker-w1-c4b379/premium-w1-c4b379-kserve-575f896d49-2n6gz"
Oct 30 20:58:31.132448 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:31.132424    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2aac3e8e-d631-4b9c-bdbb-3b4b1df133a9-home\") pod \"premium-w1-c4b379-kserve-575f896d49-2n6gz\" (UID: \"2aac3e8e-d631-4b9c-bdbb-3b4b1df133a9\") " pod="e2e-models-e2e-worker-w1-c4b379/premium-w1-c4b379-kserve-575f896d49-2n6gz"
Oct 30 20:58:31.233613 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:31.233577    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2aac3e8e-d631-4b9c-bdbb-3b4b1df133a9-tls-certs\") pod \"premium-w1-c4b379-kserve-575f896d49-2n6gz\" (UID: \"2aac3e8e-d631-4b9c-bdbb-3b4b1df133a9\") " pod="e2e-models-e2e-worker-w1-c4b379/premium-w1-c4b379-kserve-575f896d49-2n6gz"
Oct 30 20:58:31.234135 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:31.233642    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2aac3e8e-d631-4b9c-bdbb-3b4b1df133a9-model-cache\") pod \"premium-w1-c4b379-kserve-575f896d49-2n6gz\" (UID: \"2aac3e8e-d631-4b9c-bdbb-3b4b1df133a9\") " pod="e2e-models-e2e-worker-w1-c4b379/premium-w1-c4b379-kserve-575f896d49-2n6gz"
Oct 30 20:58:31.234229 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:31.234188    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2aac3e8e-d631-4b9c-bdbb-3b4b1df133a9-dshm\") pod \"premium-w1-c4b379-kserve-575f896d49-2n6gz\" (UID: \"2aac3e8e-d631-4b9c-bdbb-3b4b1df133a9\") " pod="e2e-models-e2e-worker-w1-c4b379/premium-w1-c4b379-kserve-575f896d49-2n6gz"
Oct 30 20:58:31.234288 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:31.234257    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2aac3e8e-d631-4b9c-bdbb-3b4b1df133a9-tmp-dir\") pod \"premium-w1-c4b379-kserve-575f896d49-2n6gz\" (UID: \"2aac3e8e-d631-4b9c-bdbb-3b4b1df133a9\") " pod="e2e-models-e2e-worker-w1-c4b379/premium-w1-c4b379-kserve-575f896d49-2n6gz"
Oct 30 20:58:31.234365 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:31.234343    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lrj6\" (UniqueName: \"kubernetes.io/projected/2aac3e8e-d631-4b9c-bdbb-3b4b1df133a9-kube-api-access-7lrj6\") pod \"premium-w1-c4b379-kserve-575f896d49-2n6gz\" (UID: \"2aac3e8e-d631-4b9c-bdbb-3b4b1df133a9\") " pod="e2e-models-e2e-worker-w1-c4b379/premium-w1-c4b379-kserve-575f896d49-2n6gz"
Oct 30 20:58:31.234434 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:31.234383    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2aac3e8e-d631-4b9c-bdbb-3b4b1df133a9-home\") pod \"premium-w1-c4b379-kserve-575f896d49-2n6gz\" (UID: \"2aac3e8e-d631-4b9c-bdbb-3b4b1df133a9\") " pod="e2e-models-e2e-worker-w1-c4b379/premium-w1-c4b379-kserve-575f896d49-2n6gz"
Oct 30 20:58:31.234745 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:31.234719    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2aac3e8e-d631-4b9c-bdbb-3b4b1df133a9-tmp-dir\") pod \"premium-w1-c4b379-kserve-575f896d49-2n6gz\" (UID: \"2aac3e8e-d631-4b9c-bdbb-3b4b1df133a9\") " pod="e2e-models-e2e-worker-w1-c4b379/premium-w1-c4b379-kserve-575f896d49-2n6gz"
Oct 30 20:58:31.234855 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:31.234749    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2aac3e8e-d631-4b9c-bdbb-3b4b1df133a9-home\") pod \"premium-w1-c4b379-kserve-575f896d49-2n6gz\" (UID: \"2aac3e8e-d631-4b9c-bdbb-3b4b1df133a9\") " pod="e2e-models-e2e-worker-w1-c4b379/premium-w1-c4b379-kserve-575f896d49-2n6gz"
Oct 30 20:58:31.235454 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:31.235428    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2aac3e8e-d631-4b9c-bdbb-3b4b1df133a9-model-cache\") pod \"premium-w1-c4b379-kserve-575f896d49-2n6gz\" (UID: \"2aac3e8e-d631-4b9c-bdbb-3b4b1df133a9\") " pod="e2e-models-e2e-worker-w1-c4b379/premium-w1-c4b379-kserve-575f896d49-2n6gz"
Oct 30 20:58:31.237511 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:31.237488    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2aac3e8e-d631-4b9c-bdbb-3b4b1df133a9-tls-certs\") pod \"premium-w1-c4b379-kserve-575f896d49-2n6gz\" (UID: \"2aac3e8e-d631-4b9c-bdbb-3b4b1df133a9\") " pod="e2e-models-e2e-worker-w1-c4b379/premium-w1-c4b379-kserve-575f896d49-2n6gz"
Oct 30 20:58:31.238175 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:31.238112    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2aac3e8e-d631-4b9c-bdbb-3b4b1df133a9-dshm\") pod \"premium-w1-c4b379-kserve-575f896d49-2n6gz\" (UID: \"2aac3e8e-d631-4b9c-bdbb-3b4b1df133a9\") " pod="e2e-models-e2e-worker-w1-c4b379/premium-w1-c4b379-kserve-575f896d49-2n6gz"
Oct 30 20:58:31.248061 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:31.248035    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lrj6\" (UniqueName: \"kubernetes.io/projected/2aac3e8e-d631-4b9c-bdbb-3b4b1df133a9-kube-api-access-7lrj6\") pod \"premium-w1-c4b379-kserve-575f896d49-2n6gz\" (UID: \"2aac3e8e-d631-4b9c-bdbb-3b4b1df133a9\") " pod="e2e-models-e2e-worker-w1-c4b379/premium-w1-c4b379-kserve-575f896d49-2n6gz"
Oct 30 20:58:31.308702 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:31.308668    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-c4b379/premium-w1-c4b379-kserve-575f896d49-2n6gz"
Oct 30 20:58:31.599744 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:31.599713    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-c4b379/premium-w1-c4b379-kserve-575f896d49-2n6gz"]
Oct 30 20:58:31.600038 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:58:31.600014    4521 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2aac3e8e_d631_4b9c_bdbb_3b4b1df133a9.slice/crio-f1dbafc2a63346816cf84a5612f0abb7d1e3d794c46fdb48d7c5760f638bf3c9 WatchSource:0}: Error finding container f1dbafc2a63346816cf84a5612f0abb7d1e3d794c46fdb48d7c5760f638bf3c9: Status 404 returned error can't find the container with id f1dbafc2a63346816cf84a5612f0abb7d1e3d794c46fdb48d7c5760f638bf3c9
Oct 30 20:58:31.977795 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:31.977759    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-c4b379/premium-w1-c4b379-kserve-575f896d49-2n6gz" event={"ID":"2aac3e8e-d631-4b9c-bdbb-3b4b1df133a9","Type":"ContainerStarted","Data":"58c11395b2f1ed819b5cc7f382bc5c969e7a03e693cd49102fcf67ea827d642c"}
Oct 30 20:58:31.977795 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:31.977805    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-c4b379/premium-w1-c4b379-kserve-575f896d49-2n6gz" event={"ID":"2aac3e8e-d631-4b9c-bdbb-3b4b1df133a9","Type":"ContainerStarted","Data":"f1dbafc2a63346816cf84a5612f0abb7d1e3d794c46fdb48d7c5760f638bf3c9"}
Oct 30 20:58:31.989518 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:31.989489    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5a8f94/premium-w3-5a8f94-kserve-bb96b6848-5hk8h"
Oct 30 20:58:31.997373 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:31.997308    4521 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-c4b379/premium-w1-c4b379-kserve-575f896d49-2n6gz" podStartSLOduration=1.99729497 podStartE2EDuration="1.99729497s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:58:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:58:31.996545072 +0000 UTC m=+1179.342166310" watchObservedRunningTime="2026-10-30 20:58:31.99729497 +0000 UTC m=+1179.342916208"
Oct 30 20:58:32.309217 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:32.309141    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c4b379/premium-w1-c4b379-kserve-575f896d49-2n6gz"
Oct 30 20:58:35.585013 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:35.584978    4521 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-c2d1eb/premium-w4-c2d1eb-kserve-8696fbd9b9-mxqhm"]
Oct 30 20:58:35.588815 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:35.588791    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c2d1eb/premium-w4-c2d1eb-kserve-8696fbd9b9-mxqhm"
Oct 30 20:58:35.596819 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:35.596785    4521 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-c2d1eb\"/\"default-dockercfg-mhxkm\""
Oct 30 20:58:35.596980 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:35.596920    4521 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-c2d1eb\"/\"kube-root-ca.crt\""
Oct 30 20:58:35.596980 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:35.596971    4521 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-c2d1eb\"/\"openshift-service-ca.crt\""
Oct 30 20:58:35.600109 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:35.600085    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-c2d1eb/premium-w4-c2d1eb-kserve-8696fbd9b9-mxqhm"]
Oct 30 20:58:35.600314 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:35.600299    4521 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-c2d1eb\"/\"premium-w4-c2d1eb-kserve-self-signed-certs\""
Oct 30 20:58:35.682244 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:35.682203    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4bb0f898-b77f-419d-bff4-6413e82ef51d-tmp-dir\") pod \"premium-w4-c2d1eb-kserve-8696fbd9b9-mxqhm\" (UID: \"4bb0f898-b77f-419d-bff4-6413e82ef51d\") " pod="e2e-models-e2e-worker-w4-c2d1eb/premium-w4-c2d1eb-kserve-8696fbd9b9-mxqhm"
Oct 30 20:58:35.682582 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:35.682553    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4bb0f898-b77f-419d-bff4-6413e82ef51d-model-cache\") pod \"premium-w4-c2d1eb-kserve-8696fbd9b9-mxqhm\" (UID: \"4bb0f898-b77f-419d-bff4-6413e82ef51d\") " pod="e2e-models-e2e-worker-w4-c2d1eb/premium-w4-c2d1eb-kserve-8696fbd9b9-mxqhm"
Oct 30 20:58:35.682794 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:35.682763    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4bb0f898-b77f-419d-bff4-6413e82ef51d-dshm\") pod \"premium-w4-c2d1eb-kserve-8696fbd9b9-mxqhm\" (UID: \"4bb0f898-b77f-419d-bff4-6413e82ef51d\") " pod="e2e-models-e2e-worker-w4-c2d1eb/premium-w4-c2d1eb-kserve-8696fbd9b9-mxqhm"
Oct 30 20:58:35.682879 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:35.682863    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mt6dz\" (UniqueName: \"kubernetes.io/projected/4bb0f898-b77f-419d-bff4-6413e82ef51d-kube-api-access-mt6dz\") pod \"premium-w4-c2d1eb-kserve-8696fbd9b9-mxqhm\" (UID: \"4bb0f898-b77f-419d-bff4-6413e82ef51d\") " pod="e2e-models-e2e-worker-w4-c2d1eb/premium-w4-c2d1eb-kserve-8696fbd9b9-mxqhm"
Oct 30 20:58:35.682931 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:35.682910    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4bb0f898-b77f-419d-bff4-6413e82ef51d-home\") pod \"premium-w4-c2d1eb-kserve-8696fbd9b9-mxqhm\" (UID: \"4bb0f898-b77f-419d-bff4-6413e82ef51d\") " pod="e2e-models-e2e-worker-w4-c2d1eb/premium-w4-c2d1eb-kserve-8696fbd9b9-mxqhm"
Oct 30 20:58:35.683089 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:35.683058    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4bb0f898-b77f-419d-bff4-6413e82ef51d-tls-certs\") pod \"premium-w4-c2d1eb-kserve-8696fbd9b9-mxqhm\" (UID: \"4bb0f898-b77f-419d-bff4-6413e82ef51d\") " pod="e2e-models-e2e-worker-w4-c2d1eb/premium-w4-c2d1eb-kserve-8696fbd9b9-mxqhm"
Oct 30 20:58:35.784035 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:35.784002    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4bb0f898-b77f-419d-bff4-6413e82ef51d-model-cache\") pod \"premium-w4-c2d1eb-kserve-8696fbd9b9-mxqhm\" (UID: \"4bb0f898-b77f-419d-bff4-6413e82ef51d\") " pod="e2e-models-e2e-worker-w4-c2d1eb/premium-w4-c2d1eb-kserve-8696fbd9b9-mxqhm"
Oct 30 20:58:35.784247 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:35.784046    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4bb0f898-b77f-419d-bff4-6413e82ef51d-dshm\") pod \"premium-w4-c2d1eb-kserve-8696fbd9b9-mxqhm\" (UID: \"4bb0f898-b77f-419d-bff4-6413e82ef51d\") " pod="e2e-models-e2e-worker-w4-c2d1eb/premium-w4-c2d1eb-kserve-8696fbd9b9-mxqhm"
Oct 30 20:58:35.784247 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:35.784089    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mt6dz\" (UniqueName: \"kubernetes.io/projected/4bb0f898-b77f-419d-bff4-6413e82ef51d-kube-api-access-mt6dz\") pod \"premium-w4-c2d1eb-kserve-8696fbd9b9-mxqhm\" (UID: \"4bb0f898-b77f-419d-bff4-6413e82ef51d\") " pod="e2e-models-e2e-worker-w4-c2d1eb/premium-w4-c2d1eb-kserve-8696fbd9b9-mxqhm"
Oct 30 20:58:35.784247 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:35.784113    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4bb0f898-b77f-419d-bff4-6413e82ef51d-home\") pod \"premium-w4-c2d1eb-kserve-8696fbd9b9-mxqhm\" (UID: \"4bb0f898-b77f-419d-bff4-6413e82ef51d\") " pod="e2e-models-e2e-worker-w4-c2d1eb/premium-w4-c2d1eb-kserve-8696fbd9b9-mxqhm"
Oct 30 20:58:35.784247 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:35.784143    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4bb0f898-b77f-419d-bff4-6413e82ef51d-tls-certs\") pod \"premium-w4-c2d1eb-kserve-8696fbd9b9-mxqhm\" (UID: \"4bb0f898-b77f-419d-bff4-6413e82ef51d\") " pod="e2e-models-e2e-worker-w4-c2d1eb/premium-w4-c2d1eb-kserve-8696fbd9b9-mxqhm"
Oct 30 20:58:35.784247 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:35.784192    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4bb0f898-b77f-419d-bff4-6413e82ef51d-tmp-dir\") pod \"premium-w4-c2d1eb-kserve-8696fbd9b9-mxqhm\" (UID: \"4bb0f898-b77f-419d-bff4-6413e82ef51d\") " pod="e2e-models-e2e-worker-w4-c2d1eb/premium-w4-c2d1eb-kserve-8696fbd9b9-mxqhm"
Oct 30 20:58:35.784533 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:35.784517    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4bb0f898-b77f-419d-bff4-6413e82ef51d-model-cache\") pod \"premium-w4-c2d1eb-kserve-8696fbd9b9-mxqhm\" (UID: \"4bb0f898-b77f-419d-bff4-6413e82ef51d\") " pod="e2e-models-e2e-worker-w4-c2d1eb/premium-w4-c2d1eb-kserve-8696fbd9b9-mxqhm"
Oct 30 20:58:35.784974 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:35.784945    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4bb0f898-b77f-419d-bff4-6413e82ef51d-home\") pod \"premium-w4-c2d1eb-kserve-8696fbd9b9-mxqhm\" (UID: \"4bb0f898-b77f-419d-bff4-6413e82ef51d\") " pod="e2e-models-e2e-worker-w4-c2d1eb/premium-w4-c2d1eb-kserve-8696fbd9b9-mxqhm"
Oct 30 20:58:35.785125 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:35.785105    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4bb0f898-b77f-419d-bff4-6413e82ef51d-tmp-dir\") pod \"premium-w4-c2d1eb-kserve-8696fbd9b9-mxqhm\" (UID: \"4bb0f898-b77f-419d-bff4-6413e82ef51d\") " pod="e2e-models-e2e-worker-w4-c2d1eb/premium-w4-c2d1eb-kserve-8696fbd9b9-mxqhm"
Oct 30 20:58:35.786681 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:35.786650    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4bb0f898-b77f-419d-bff4-6413e82ef51d-dshm\") pod \"premium-w4-c2d1eb-kserve-8696fbd9b9-mxqhm\" (UID: \"4bb0f898-b77f-419d-bff4-6413e82ef51d\") " pod="e2e-models-e2e-worker-w4-c2d1eb/premium-w4-c2d1eb-kserve-8696fbd9b9-mxqhm"
Oct 30 20:58:35.787085 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:35.787061    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4bb0f898-b77f-419d-bff4-6413e82ef51d-tls-certs\") pod \"premium-w4-c2d1eb-kserve-8696fbd9b9-mxqhm\" (UID: \"4bb0f898-b77f-419d-bff4-6413e82ef51d\") " pod="e2e-models-e2e-worker-w4-c2d1eb/premium-w4-c2d1eb-kserve-8696fbd9b9-mxqhm"
Oct 30 20:58:35.797203 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:35.797168    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mt6dz\" (UniqueName: \"kubernetes.io/projected/4bb0f898-b77f-419d-bff4-6413e82ef51d-kube-api-access-mt6dz\") pod \"premium-w4-c2d1eb-kserve-8696fbd9b9-mxqhm\" (UID: \"4bb0f898-b77f-419d-bff4-6413e82ef51d\") " pod="e2e-models-e2e-worker-w4-c2d1eb/premium-w4-c2d1eb-kserve-8696fbd9b9-mxqhm"
Oct 30 20:58:35.915418 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:35.915309    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c2d1eb/premium-w4-c2d1eb-kserve-8696fbd9b9-mxqhm"
Oct 30 20:58:36.220709 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:36.220678    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-c2d1eb/premium-w4-c2d1eb-kserve-8696fbd9b9-mxqhm"]
Oct 30 20:58:36.222361 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:58:36.222319    4521 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4bb0f898_b77f_419d_bff4_6413e82ef51d.slice/crio-6e0b81cd2e63852db6b987dae98cfeaddecd229881b780ebd6f64910abbccaab WatchSource:0}: Error finding container 6e0b81cd2e63852db6b987dae98cfeaddecd229881b780ebd6f64910abbccaab: Status 404 returned error can't find the container with id 6e0b81cd2e63852db6b987dae98cfeaddecd229881b780ebd6f64910abbccaab
Oct 30 20:58:37.016343 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:37.016289    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c2d1eb/premium-w4-c2d1eb-kserve-8696fbd9b9-mxqhm" event={"ID":"4bb0f898-b77f-419d-bff4-6413e82ef51d","Type":"ContainerStarted","Data":"cf06834ad798b61733cac66d7babac65e2e8816c2cc87cdfd0eba97e9cd9d770"}
Oct 30 20:58:37.016811 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:37.016357    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c2d1eb/premium-w4-c2d1eb-kserve-8696fbd9b9-mxqhm" event={"ID":"4bb0f898-b77f-419d-bff4-6413e82ef51d","Type":"ContainerStarted","Data":"6e0b81cd2e63852db6b987dae98cfeaddecd229881b780ebd6f64910abbccaab"}
Oct 30 20:58:37.038953 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:37.038897    4521 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-c2d1eb/premium-w4-c2d1eb-kserve-8696fbd9b9-mxqhm" podStartSLOduration=2.038883999 podStartE2EDuration="2.038883999s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:58:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:58:37.037541963 +0000 UTC m=+1184.383163200" watchObservedRunningTime="2026-10-30 20:58:37.038883999 +0000 UTC m=+1184.384505238"
Oct 30 20:58:37.811341 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:37.811294    4521 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-65968b/premium-w5-65968b-kserve-67977485f7-c9jpv"
Oct 30 20:58:37.835513 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:37.835483    4521 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-65968b/premium-w5-65968b-kserve-67977485f7-c9jpv"
Oct 30 20:58:37.916317 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:37.916273    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c2d1eb/premium-w4-c2d1eb-kserve-8696fbd9b9-mxqhm"
Oct 30 20:58:38.034897 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:38.034868    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-65968b/premium-w5-65968b-kserve-67977485f7-c9jpv"
Oct 30 20:58:38.522772 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:38.522739    4521 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-39fb96/premium-w6-39fb96-kserve-b9cf7f75c-96blx"]
Oct 30 20:58:38.528242 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:38.528211    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-39fb96/premium-w6-39fb96-kserve-b9cf7f75c-96blx"
Oct 30 20:58:38.533120 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:38.533092    4521 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-39fb96\"/\"openshift-service-ca.crt\""
Oct 30 20:58:38.533872 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:38.533850    4521 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-39fb96\"/\"kube-root-ca.crt\""
Oct 30 20:58:38.534489 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:38.534463    4521 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-39fb96\"/\"default-dockercfg-flj6l\""
Oct 30 20:58:38.534576 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:38.534555    4521 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-39fb96\"/\"premium-w6-39fb96-kserve-self-signed-certs\""
Oct 30 20:58:38.540214 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:38.540188    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-39fb96/premium-w6-39fb96-kserve-b9cf7f75c-96blx"]
Oct 30 20:58:38.716417 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:38.716365    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4bbk\" (UniqueName: \"kubernetes.io/projected/18ca0a94-c322-41fa-9fd9-0f14419b5075-kube-api-access-j4bbk\") pod \"premium-w6-39fb96-kserve-b9cf7f75c-96blx\" (UID: \"18ca0a94-c322-41fa-9fd9-0f14419b5075\") " pod="e2e-models-e2e-worker-w6-39fb96/premium-w6-39fb96-kserve-b9cf7f75c-96blx"
Oct 30 20:58:38.716583 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:38.716436    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18ca0a94-c322-41fa-9fd9-0f14419b5075-dshm\") pod \"premium-w6-39fb96-kserve-b9cf7f75c-96blx\" (UID: \"18ca0a94-c322-41fa-9fd9-0f14419b5075\") " pod="e2e-models-e2e-worker-w6-39fb96/premium-w6-39fb96-kserve-b9cf7f75c-96blx"
Oct 30 20:58:38.716583 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:38.716535    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18ca0a94-c322-41fa-9fd9-0f14419b5075-tls-certs\") pod \"premium-w6-39fb96-kserve-b9cf7f75c-96blx\" (UID: \"18ca0a94-c322-41fa-9fd9-0f14419b5075\") " pod="e2e-models-e2e-worker-w6-39fb96/premium-w6-39fb96-kserve-b9cf7f75c-96blx"
Oct 30 20:58:38.716709 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:38.716579    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18ca0a94-c322-41fa-9fd9-0f14419b5075-tmp-dir\") pod \"premium-w6-39fb96-kserve-b9cf7f75c-96blx\" (UID: \"18ca0a94-c322-41fa-9fd9-0f14419b5075\") " pod="e2e-models-e2e-worker-w6-39fb96/premium-w6-39fb96-kserve-b9cf7f75c-96blx"
Oct 30 20:58:38.716709 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:38.716622    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18ca0a94-c322-41fa-9fd9-0f14419b5075-home\") pod \"premium-w6-39fb96-kserve-b9cf7f75c-96blx\" (UID: \"18ca0a94-c322-41fa-9fd9-0f14419b5075\") " pod="e2e-models-e2e-worker-w6-39fb96/premium-w6-39fb96-kserve-b9cf7f75c-96blx"
Oct 30 20:58:38.716792 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:38.716744    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18ca0a94-c322-41fa-9fd9-0f14419b5075-model-cache\") pod \"premium-w6-39fb96-kserve-b9cf7f75c-96blx\" (UID: \"18ca0a94-c322-41fa-9fd9-0f14419b5075\") " pod="e2e-models-e2e-worker-w6-39fb96/premium-w6-39fb96-kserve-b9cf7f75c-96blx"
Oct 30 20:58:38.818245 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:38.818150    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18ca0a94-c322-41fa-9fd9-0f14419b5075-model-cache\") pod \"premium-w6-39fb96-kserve-b9cf7f75c-96blx\" (UID: \"18ca0a94-c322-41fa-9fd9-0f14419b5075\") " pod="e2e-models-e2e-worker-w6-39fb96/premium-w6-39fb96-kserve-b9cf7f75c-96blx"
Oct 30 20:58:38.818439 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:38.818292    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4bbk\" (UniqueName: \"kubernetes.io/projected/18ca0a94-c322-41fa-9fd9-0f14419b5075-kube-api-access-j4bbk\") pod \"premium-w6-39fb96-kserve-b9cf7f75c-96blx\" (UID: \"18ca0a94-c322-41fa-9fd9-0f14419b5075\") " pod="e2e-models-e2e-worker-w6-39fb96/premium-w6-39fb96-kserve-b9cf7f75c-96blx"
Oct 30 20:58:38.818439 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:38.818350    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18ca0a94-c322-41fa-9fd9-0f14419b5075-dshm\") pod \"premium-w6-39fb96-kserve-b9cf7f75c-96blx\" (UID: \"18ca0a94-c322-41fa-9fd9-0f14419b5075\") " pod="e2e-models-e2e-worker-w6-39fb96/premium-w6-39fb96-kserve-b9cf7f75c-96blx"
Oct 30 20:58:38.818439 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:38.818380    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18ca0a94-c322-41fa-9fd9-0f14419b5075-tls-certs\") pod \"premium-w6-39fb96-kserve-b9cf7f75c-96blx\" (UID: \"18ca0a94-c322-41fa-9fd9-0f14419b5075\") " pod="e2e-models-e2e-worker-w6-39fb96/premium-w6-39fb96-kserve-b9cf7f75c-96blx"
Oct 30 20:58:38.818439 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:38.818408    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18ca0a94-c322-41fa-9fd9-0f14419b5075-tmp-dir\") pod \"premium-w6-39fb96-kserve-b9cf7f75c-96blx\" (UID: \"18ca0a94-c322-41fa-9fd9-0f14419b5075\") " pod="e2e-models-e2e-worker-w6-39fb96/premium-w6-39fb96-kserve-b9cf7f75c-96blx"
Oct 30 20:58:38.818658 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:38.818443    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18ca0a94-c322-41fa-9fd9-0f14419b5075-home\") pod \"premium-w6-39fb96-kserve-b9cf7f75c-96blx\" (UID: \"18ca0a94-c322-41fa-9fd9-0f14419b5075\") " pod="e2e-models-e2e-worker-w6-39fb96/premium-w6-39fb96-kserve-b9cf7f75c-96blx"
Oct 30 20:58:38.818765 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:38.818742    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18ca0a94-c322-41fa-9fd9-0f14419b5075-model-cache\") pod \"premium-w6-39fb96-kserve-b9cf7f75c-96blx\" (UID: \"18ca0a94-c322-41fa-9fd9-0f14419b5075\") " pod="e2e-models-e2e-worker-w6-39fb96/premium-w6-39fb96-kserve-b9cf7f75c-96blx"
Oct 30 20:58:38.818884 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:38.818857    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18ca0a94-c322-41fa-9fd9-0f14419b5075-home\") pod \"premium-w6-39fb96-kserve-b9cf7f75c-96blx\" (UID: \"18ca0a94-c322-41fa-9fd9-0f14419b5075\") " pod="e2e-models-e2e-worker-w6-39fb96/premium-w6-39fb96-kserve-b9cf7f75c-96blx"
Oct 30 20:58:38.819259 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:38.819239    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18ca0a94-c322-41fa-9fd9-0f14419b5075-tmp-dir\") pod \"premium-w6-39fb96-kserve-b9cf7f75c-96blx\" (UID: \"18ca0a94-c322-41fa-9fd9-0f14419b5075\") " pod="e2e-models-e2e-worker-w6-39fb96/premium-w6-39fb96-kserve-b9cf7f75c-96blx"
Oct 30 20:58:38.821464 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:38.821433    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18ca0a94-c322-41fa-9fd9-0f14419b5075-dshm\") pod \"premium-w6-39fb96-kserve-b9cf7f75c-96blx\" (UID: \"18ca0a94-c322-41fa-9fd9-0f14419b5075\") " pod="e2e-models-e2e-worker-w6-39fb96/premium-w6-39fb96-kserve-b9cf7f75c-96blx"
Oct 30 20:58:38.821637 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:38.821602    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18ca0a94-c322-41fa-9fd9-0f14419b5075-tls-certs\") pod \"premium-w6-39fb96-kserve-b9cf7f75c-96blx\" (UID: \"18ca0a94-c322-41fa-9fd9-0f14419b5075\") " pod="e2e-models-e2e-worker-w6-39fb96/premium-w6-39fb96-kserve-b9cf7f75c-96blx"
Oct 30 20:58:38.831298 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:38.831269    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4bbk\" (UniqueName: \"kubernetes.io/projected/18ca0a94-c322-41fa-9fd9-0f14419b5075-kube-api-access-j4bbk\") pod \"premium-w6-39fb96-kserve-b9cf7f75c-96blx\" (UID: \"18ca0a94-c322-41fa-9fd9-0f14419b5075\") " pod="e2e-models-e2e-worker-w6-39fb96/premium-w6-39fb96-kserve-b9cf7f75c-96blx"
Oct 30 20:58:38.844074 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:38.844042    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-39fb96/premium-w6-39fb96-kserve-b9cf7f75c-96blx"
Oct 30 20:58:39.129141 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:39.129117    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-39fb96/premium-w6-39fb96-kserve-b9cf7f75c-96blx"]
Oct 30 20:58:39.131129 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:58:39.131099    4521 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod18ca0a94_c322_41fa_9fd9_0f14419b5075.slice/crio-6071fcf4e071d65a907c727159a694ae100b0ce9c4482a0f4d9c467fe33920ba WatchSource:0}: Error finding container 6071fcf4e071d65a907c727159a694ae100b0ce9c4482a0f4d9c467fe33920ba: Status 404 returned error can't find the container with id 6071fcf4e071d65a907c727159a694ae100b0ce9c4482a0f4d9c467fe33920ba
Oct 30 20:58:40.039340 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:40.039284    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-39fb96/premium-w6-39fb96-kserve-b9cf7f75c-96blx" event={"ID":"18ca0a94-c322-41fa-9fd9-0f14419b5075","Type":"ContainerStarted","Data":"e70b88f888be048aa9ffe8bbc08b0dcc8c03d1e8c89116f6df4e0a302aeb05ff"}
Oct 30 20:58:40.039561 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:40.039366    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-39fb96/premium-w6-39fb96-kserve-b9cf7f75c-96blx" event={"ID":"18ca0a94-c322-41fa-9fd9-0f14419b5075","Type":"ContainerStarted","Data":"6071fcf4e071d65a907c727159a694ae100b0ce9c4482a0f4d9c467fe33920ba"}
Oct 30 20:58:40.059585 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:40.059533    4521 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-39fb96/premium-w6-39fb96-kserve-b9cf7f75c-96blx" podStartSLOduration=2.059518661 podStartE2EDuration="2.059518661s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:58:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:58:40.058777765 +0000 UTC m=+1187.404399003" watchObservedRunningTime="2026-10-30 20:58:40.059518661 +0000 UTC m=+1187.405139938"
Oct 30 20:58:40.844791 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:40.844757    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-39fb96/premium-w6-39fb96-kserve-b9cf7f75c-96blx"
Oct 30 20:58:41.309818 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:41.309781    4521 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-c4b379/premium-w1-c4b379-kserve-575f896d49-2n6gz"
Oct 30 20:58:41.323494 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:41.323462    4521 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-c4b379/premium-w1-c4b379-kserve-575f896d49-2n6gz"
Oct 30 20:58:42.066825 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:42.066797    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c4b379/premium-w1-c4b379-kserve-575f896d49-2n6gz"
Oct 30 20:58:43.958809 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:43.958774    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4kxb"]
Oct 30 20:58:45.915817 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:45.915781    4521 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-c2d1eb/premium-w4-c2d1eb-kserve-8696fbd9b9-mxqhm"
Oct 30 20:58:45.933959 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:45.933932    4521 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-c2d1eb/premium-w4-c2d1eb-kserve-8696fbd9b9-mxqhm"
Oct 30 20:58:46.088102 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:46.088070    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c2d1eb/premium-w4-c2d1eb-kserve-8696fbd9b9-mxqhm"
Oct 30 20:58:46.431107 ip-10-0-131-125 kubenswrapper[4521]: E1030 20:58:46.431068    4521 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lkn2k" containerName="extractor"
Oct 30 20:58:48.845211 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:48.845160    4521 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-39fb96/premium-w6-39fb96-kserve-b9cf7f75c-96blx"
Oct 30 20:58:48.858016 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:48.857978    4521 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-39fb96/premium-w6-39fb96-kserve-b9cf7f75c-96blx"
Oct 30 20:58:49.108494 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:58:49.108413    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-39fb96/premium-w6-39fb96-kserve-b9cf7f75c-96blx"
Oct 30 20:59:16.430880 ip-10-0-131-125 kubenswrapper[4521]: E1030 20:59:16.430794    4521 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lkn2k" containerName="extractor"
Oct 30 20:59:16.491180 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:16.491148    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4kxb"]
Oct 30 20:59:18.284100 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:18.284055    4521 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52c565cc-f2ppx"]
Oct 30 20:59:18.287798 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:18.287774    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52c565cc-f2ppx"
Oct 30 20:59:18.293925 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:18.293387    4521 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:59:18.294647 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:18.294621    4521 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c45pw\""
Oct 30 20:59:18.298115 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:18.298092    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52c565cc-f2ppx"]
Oct 30 20:59:18.336344 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:18.336264    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szmw9\" (UniqueName: \"kubernetes.io/projected/06e939c1-2403-439d-b599-ab8ff61a042d-kube-api-access-szmw9\") pod \"maas-api-revoke-sub-52c565cc-f2ppx\" (UID: \"06e939c1-2403-439d-b599-ab8ff61a042d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52c565cc-f2ppx"
Oct 30 20:59:18.336525 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:18.336420    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06e939c1-2403-439d-b599-ab8ff61a042d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-52c565cc-f2ppx\" (UID: \"06e939c1-2403-439d-b599-ab8ff61a042d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52c565cc-f2ppx"
Oct 30 20:59:18.336525 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:18.336477    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06e939c1-2403-439d-b599-ab8ff61a042d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-52c565cc-f2ppx\" (UID: \"06e939c1-2403-439d-b599-ab8ff61a042d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52c565cc-f2ppx"
Oct 30 20:59:18.437844 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:18.437807    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szmw9\" (UniqueName: \"kubernetes.io/projected/06e939c1-2403-439d-b599-ab8ff61a042d-kube-api-access-szmw9\") pod \"maas-api-revoke-sub-52c565cc-f2ppx\" (UID: \"06e939c1-2403-439d-b599-ab8ff61a042d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52c565cc-f2ppx"
Oct 30 20:59:18.438046 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:18.437856    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06e939c1-2403-439d-b599-ab8ff61a042d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-52c565cc-f2ppx\" (UID: \"06e939c1-2403-439d-b599-ab8ff61a042d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52c565cc-f2ppx"
Oct 30 20:59:18.438046 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:18.437903    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06e939c1-2403-439d-b599-ab8ff61a042d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-52c565cc-f2ppx\" (UID: \"06e939c1-2403-439d-b599-ab8ff61a042d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52c565cc-f2ppx"
Oct 30 20:59:18.438730 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:18.438704    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06e939c1-2403-439d-b599-ab8ff61a042d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-52c565cc-f2ppx\" (UID: \"06e939c1-2403-439d-b599-ab8ff61a042d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52c565cc-f2ppx"
Oct 30 20:59:18.440372 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:18.440343    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06e939c1-2403-439d-b599-ab8ff61a042d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-52c565cc-f2ppx\" (UID: \"06e939c1-2403-439d-b599-ab8ff61a042d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52c565cc-f2ppx"
Oct 30 20:59:18.450377 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:18.450354    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-szmw9\" (UniqueName: \"kubernetes.io/projected/06e939c1-2403-439d-b599-ab8ff61a042d-kube-api-access-szmw9\") pod \"maas-api-revoke-sub-52c565cc-f2ppx\" (UID: \"06e939c1-2403-439d-b599-ab8ff61a042d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52c565cc-f2ppx"
Oct 30 20:59:18.602604 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:18.602524    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52c565cc-f2ppx"
Oct 30 20:59:18.882133 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:18.882103    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52c565cc-f2ppx"]
Oct 30 20:59:18.884072 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:59:18.884043    4521 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06e939c1_2403_439d_b599_ab8ff61a042d.slice/crio-be2194e3761f5edd68f4e3539d7fd172c161b99a262634b40928d08ba78e4d32 WatchSource:0}: Error finding container be2194e3761f5edd68f4e3539d7fd172c161b99a262634b40928d08ba78e4d32: Status 404 returned error can't find the container with id be2194e3761f5edd68f4e3539d7fd172c161b99a262634b40928d08ba78e4d32
Oct 30 20:59:18.885924 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:18.885906    4521 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 20:59:19.274620 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:19.274582    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52c565cc-f2ppx" event={"ID":"06e939c1-2403-439d-b599-ab8ff61a042d","Type":"ContainerStarted","Data":"be2194e3761f5edd68f4e3539d7fd172c161b99a262634b40928d08ba78e4d32"}
Oct 30 20:59:22.301105 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:22.301063    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52c565cc-f2ppx" event={"ID":"06e939c1-2403-439d-b599-ab8ff61a042d","Type":"ContainerStarted","Data":"daa972aea978ae6626914a284ac2ec68cf2930b5f5590660d25a7420cd5df4d5"}
Oct 30 20:59:22.320807 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:22.320751    4521 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52c565cc-f2ppx" podStartSLOduration=1.7334938260000001 podStartE2EDuration="4.320736171s" totalImagesPullingTime="2.587242345s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:59:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:59:22.318739689 +0000 UTC m=+1229.664360930" watchObservedRunningTime="2026-10-30 20:59:22.320736171 +0000 UTC m=+1229.666357408"
Oct 30 20:59:23.307846 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:23.307810    4521 generic.go:356] "Generic (PLEG): container finished" podID="06e939c1-2403-439d-b599-ab8ff61a042d" containerID="daa972aea978ae6626914a284ac2ec68cf2930b5f5590660d25a7420cd5df4d5" exitCode=0
Oct 30 20:59:23.308274 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:23.307907    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52c565cc-f2ppx" event={"ID":"06e939c1-2403-439d-b599-ab8ff61a042d","Type":"ContainerDied","Data":"daa972aea978ae6626914a284ac2ec68cf2930b5f5590660d25a7420cd5df4d5"}
Oct 30 20:59:24.462904 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:24.462865    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52c565cc-f2ppx"
Oct 30 20:59:24.503372 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:24.503336    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06e939c1-2403-439d-b599-ab8ff61a042d-kube-api-access-szmw9\" (UniqueName: \"kubernetes.io/projected/06e939c1-2403-439d-b599-ab8ff61a042d-kube-api-access-szmw9\") pod \"06e939c1-2403-439d-b599-ab8ff61a042d\" (UID: \"06e939c1-2403-439d-b599-ab8ff61a042d\") "
Oct 30 20:59:24.503555 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:24.503395    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/06e939c1-2403-439d-b599-ab8ff61a042d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06e939c1-2403-439d-b599-ab8ff61a042d-maas-api-xks-ca\") pod \"06e939c1-2403-439d-b599-ab8ff61a042d\" (UID: \"06e939c1-2403-439d-b599-ab8ff61a042d\") "
Oct 30 20:59:24.503555 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:24.503425    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06e939c1-2403-439d-b599-ab8ff61a042d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06e939c1-2403-439d-b599-ab8ff61a042d-maas-api-openshift-ca\") pod \"06e939c1-2403-439d-b599-ab8ff61a042d\" (UID: \"06e939c1-2403-439d-b599-ab8ff61a042d\") "
Oct 30 20:59:24.503863 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:24.503834    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06e939c1-2403-439d-b599-ab8ff61a042d-maas-api-openshift-ca" pod "06e939c1-2403-439d-b599-ab8ff61a042d" (UID: "06e939c1-2403-439d-b599-ab8ff61a042d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:59:24.505631 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:24.505597    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06e939c1-2403-439d-b599-ab8ff61a042d-maas-api-xks-ca" pod "06e939c1-2403-439d-b599-ab8ff61a042d" (UID: "06e939c1-2403-439d-b599-ab8ff61a042d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:59:24.505631 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:24.505618    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06e939c1-2403-439d-b599-ab8ff61a042d-kube-api-access-szmw9" pod "06e939c1-2403-439d-b599-ab8ff61a042d" (UID: "06e939c1-2403-439d-b599-ab8ff61a042d"). InnerVolumeSpecName "kube-api-access-szmw9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:59:24.604923 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:24.604840    4521 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szmw9\" (UniqueName: \"kubernetes.io/projected/06e939c1-2403-439d-b599-ab8ff61a042d-kube-api-access-szmw9\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:59:24.604923 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:24.604874    4521 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06e939c1-2403-439d-b599-ab8ff61a042d-maas-api-xks-ca\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:59:24.604923 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:24.604885    4521 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06e939c1-2403-439d-b599-ab8ff61a042d-maas-api-openshift-ca\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:59:25.322690 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:25.322656    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52c565cc-f2ppx"
Oct 30 20:59:25.322690 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:25.322668    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52c565cc-f2ppx" event={"ID":"06e939c1-2403-439d-b599-ab8ff61a042d","Type":"ContainerDied","Data":"be2194e3761f5edd68f4e3539d7fd172c161b99a262634b40928d08ba78e4d32"}
Oct 30 20:59:25.322954 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:25.322715    4521 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="be2194e3761f5edd68f4e3539d7fd172c161b99a262634b40928d08ba78e4d32"
Oct 30 20:59:30.833645 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:30.833605    4521 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf7689f5-sjgsr"]
Oct 30 20:59:30.834248 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:30.834228    4521 cpu_manager.go:414] "containerMap: removing container" podUID="06e939c1-2403-439d-b599-ab8ff61a042d" containerName="revoke-keys"
Oct 30 20:59:30.834312 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:30.834250    4521 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06e939c1-2403-439d-b599-ab8ff61a042d" containerName="revoke-keys"
Oct 30 20:59:30.834407 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:30.834394    4521 memory_manager.go:365] "RemoveStaleState removing state" podUID="06e939c1-2403-439d-b599-ab8ff61a042d" containerName="revoke-keys"
Oct 30 20:59:30.838271 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:30.838246    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf7689f5-sjgsr"
Oct 30 20:59:30.843242 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:30.843220    4521 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:59:30.843446 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:30.843275    4521 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c45pw\""
Oct 30 20:59:30.845901 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:30.845880    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf7689f5-sjgsr"]
Oct 30 20:59:30.971152 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:30.971117    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gb5j\" (UniqueName: \"kubernetes.io/projected/82e161c9-b3d1-4448-9cd9-403a0b6ffa3a-kube-api-access-8gb5j\") pod \"maas-api-revoke-sub-bf7689f5-sjgsr\" (UID: \"82e161c9-b3d1-4448-9cd9-403a0b6ffa3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf7689f5-sjgsr"
Oct 30 20:59:30.971351 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:30.971165    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82e161c9-b3d1-4448-9cd9-403a0b6ffa3a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf7689f5-sjgsr\" (UID: \"82e161c9-b3d1-4448-9cd9-403a0b6ffa3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf7689f5-sjgsr"
Oct 30 20:59:30.971351 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:30.971302    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82e161c9-b3d1-4448-9cd9-403a0b6ffa3a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf7689f5-sjgsr\" (UID: \"82e161c9-b3d1-4448-9cd9-403a0b6ffa3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf7689f5-sjgsr"
Oct 30 20:59:31.072601 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:31.072552    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82e161c9-b3d1-4448-9cd9-403a0b6ffa3a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf7689f5-sjgsr\" (UID: \"82e161c9-b3d1-4448-9cd9-403a0b6ffa3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf7689f5-sjgsr"
Oct 30 20:59:31.072795 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:31.072692    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gb5j\" (UniqueName: \"kubernetes.io/projected/82e161c9-b3d1-4448-9cd9-403a0b6ffa3a-kube-api-access-8gb5j\") pod \"maas-api-revoke-sub-bf7689f5-sjgsr\" (UID: \"82e161c9-b3d1-4448-9cd9-403a0b6ffa3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf7689f5-sjgsr"
Oct 30 20:59:31.072795 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:31.072745    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82e161c9-b3d1-4448-9cd9-403a0b6ffa3a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf7689f5-sjgsr\" (UID: \"82e161c9-b3d1-4448-9cd9-403a0b6ffa3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf7689f5-sjgsr"
Oct 30 20:59:31.073389 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:31.073361    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82e161c9-b3d1-4448-9cd9-403a0b6ffa3a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf7689f5-sjgsr\" (UID: \"82e161c9-b3d1-4448-9cd9-403a0b6ffa3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf7689f5-sjgsr"
Oct 30 20:59:31.075339 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:31.075298    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82e161c9-b3d1-4448-9cd9-403a0b6ffa3a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf7689f5-sjgsr\" (UID: \"82e161c9-b3d1-4448-9cd9-403a0b6ffa3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf7689f5-sjgsr"
Oct 30 20:59:31.086035 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:31.085968    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gb5j\" (UniqueName: \"kubernetes.io/projected/82e161c9-b3d1-4448-9cd9-403a0b6ffa3a-kube-api-access-8gb5j\") pod \"maas-api-revoke-sub-bf7689f5-sjgsr\" (UID: \"82e161c9-b3d1-4448-9cd9-403a0b6ffa3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf7689f5-sjgsr"
Oct 30 20:59:31.155158 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:31.155119    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf7689f5-sjgsr"
Oct 30 20:59:31.440064 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:31.440031    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf7689f5-sjgsr"]
Oct 30 20:59:31.440200 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:59:31.440143    4521 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82e161c9_b3d1_4448_9cd9_403a0b6ffa3a.slice/crio-10ae5f38786f7f0a60d46885c276c4a5365d3b507ecc443908747a93717693a4 WatchSource:0}: Error finding container 10ae5f38786f7f0a60d46885c276c4a5365d3b507ecc443908747a93717693a4: Status 404 returned error can't find the container with id 10ae5f38786f7f0a60d46885c276c4a5365d3b507ecc443908747a93717693a4
Oct 30 20:59:32.368807 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:32.368767    4521 generic.go:356] "Generic (PLEG): container finished" podID="82e161c9-b3d1-4448-9cd9-403a0b6ffa3a" containerID="e20eda67237e3bb82f232e8d05704f547e15efe2dc1cf956bab94bc206775893" exitCode=0
Oct 30 20:59:32.369186 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:32.368842    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf7689f5-sjgsr" event={"ID":"82e161c9-b3d1-4448-9cd9-403a0b6ffa3a","Type":"ContainerDied","Data":"e20eda67237e3bb82f232e8d05704f547e15efe2dc1cf956bab94bc206775893"}
Oct 30 20:59:32.369186 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:32.368900    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf7689f5-sjgsr" event={"ID":"82e161c9-b3d1-4448-9cd9-403a0b6ffa3a","Type":"ContainerStarted","Data":"10ae5f38786f7f0a60d46885c276c4a5365d3b507ecc443908747a93717693a4"}
Oct 30 20:59:32.801910 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:32.801874    4521 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11299c87-dlggc"]
Oct 30 20:59:32.806056 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:32.806030    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11299c87-dlggc"]
Oct 30 20:59:32.806227 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:32.806152    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11299c87-dlggc"
Oct 30 20:59:32.893398 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:32.893359    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5jmx\" (UniqueName: \"kubernetes.io/projected/eecab7a4-8911-4b5f-b6fa-97727ae3e1cd-kube-api-access-z5jmx\") pod \"maas-api-revoke-sub-11299c87-dlggc\" (UID: \"eecab7a4-8911-4b5f-b6fa-97727ae3e1cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11299c87-dlggc"
Oct 30 20:59:32.893587 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:32.893416    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eecab7a4-8911-4b5f-b6fa-97727ae3e1cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-11299c87-dlggc\" (UID: \"eecab7a4-8911-4b5f-b6fa-97727ae3e1cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11299c87-dlggc"
Oct 30 20:59:32.893587 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:32.893547    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eecab7a4-8911-4b5f-b6fa-97727ae3e1cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-11299c87-dlggc\" (UID: \"eecab7a4-8911-4b5f-b6fa-97727ae3e1cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11299c87-dlggc"
Oct 30 20:59:32.994678 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:32.994637    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5jmx\" (UniqueName: \"kubernetes.io/projected/eecab7a4-8911-4b5f-b6fa-97727ae3e1cd-kube-api-access-z5jmx\") pod \"maas-api-revoke-sub-11299c87-dlggc\" (UID: \"eecab7a4-8911-4b5f-b6fa-97727ae3e1cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11299c87-dlggc"
Oct 30 20:59:32.994678 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:32.994683    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eecab7a4-8911-4b5f-b6fa-97727ae3e1cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-11299c87-dlggc\" (UID: \"eecab7a4-8911-4b5f-b6fa-97727ae3e1cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11299c87-dlggc"
Oct 30 20:59:32.994889 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:32.994734    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eecab7a4-8911-4b5f-b6fa-97727ae3e1cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-11299c87-dlggc\" (UID: \"eecab7a4-8911-4b5f-b6fa-97727ae3e1cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11299c87-dlggc"
Oct 30 20:59:32.995587 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:32.995562    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eecab7a4-8911-4b5f-b6fa-97727ae3e1cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-11299c87-dlggc\" (UID: \"eecab7a4-8911-4b5f-b6fa-97727ae3e1cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11299c87-dlggc"
Oct 30 20:59:32.997186 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:32.997153    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eecab7a4-8911-4b5f-b6fa-97727ae3e1cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-11299c87-dlggc\" (UID: \"eecab7a4-8911-4b5f-b6fa-97727ae3e1cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11299c87-dlggc"
Oct 30 20:59:33.007747 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:33.007716    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5jmx\" (UniqueName: \"kubernetes.io/projected/eecab7a4-8911-4b5f-b6fa-97727ae3e1cd-kube-api-access-z5jmx\") pod \"maas-api-revoke-sub-11299c87-dlggc\" (UID: \"eecab7a4-8911-4b5f-b6fa-97727ae3e1cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11299c87-dlggc"
Oct 30 20:59:33.122268 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:33.122143    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11299c87-dlggc"
Oct 30 20:59:33.411619 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:33.411592    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11299c87-dlggc"]
Oct 30 20:59:33.413710 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:59:33.413680    4521 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeecab7a4_8911_4b5f_b6fa_97727ae3e1cd.slice/crio-aa6ff878cbe28b13e0877ed5ff1d31020ba78aa2ce52faf2503ee0429b688ebd WatchSource:0}: Error finding container aa6ff878cbe28b13e0877ed5ff1d31020ba78aa2ce52faf2503ee0429b688ebd: Status 404 returned error can't find the container with id aa6ff878cbe28b13e0877ed5ff1d31020ba78aa2ce52faf2503ee0429b688ebd
Oct 30 20:59:33.516880 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:33.516850    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf7689f5-sjgsr"
Oct 30 20:59:33.565214 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:33.565179    4521 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d92a4e2c-mvb6w"]
Oct 30 20:59:33.565644 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:33.565630    4521 cpu_manager.go:414] "containerMap: removing container" podUID="82e161c9-b3d1-4448-9cd9-403a0b6ffa3a" containerName="revoke-keys"
Oct 30 20:59:33.565644 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:33.565644    4521 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82e161c9-b3d1-4448-9cd9-403a0b6ffa3a" containerName="revoke-keys"
Oct 30 20:59:33.565766 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:33.565720    4521 memory_manager.go:365] "RemoveStaleState removing state" podUID="82e161c9-b3d1-4448-9cd9-403a0b6ffa3a" containerName="revoke-keys"
Oct 30 20:59:33.568889 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:33.568864    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d92a4e2c-mvb6w"
Oct 30 20:59:33.577428 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:33.577174    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d92a4e2c-mvb6w"]
Oct 30 20:59:33.600537 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:33.600507    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/82e161c9-b3d1-4448-9cd9-403a0b6ffa3a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82e161c9-b3d1-4448-9cd9-403a0b6ffa3a-maas-api-xks-ca\") pod \"82e161c9-b3d1-4448-9cd9-403a0b6ffa3a\" (UID: \"82e161c9-b3d1-4448-9cd9-403a0b6ffa3a\") "
Oct 30 20:59:33.600724 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:33.600576    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82e161c9-b3d1-4448-9cd9-403a0b6ffa3a-kube-api-access-8gb5j\" (UniqueName: \"kubernetes.io/projected/82e161c9-b3d1-4448-9cd9-403a0b6ffa3a-kube-api-access-8gb5j\") pod \"82e161c9-b3d1-4448-9cd9-403a0b6ffa3a\" (UID: \"82e161c9-b3d1-4448-9cd9-403a0b6ffa3a\") "
Oct 30 20:59:33.600724 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:33.600718    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82e161c9-b3d1-4448-9cd9-403a0b6ffa3a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82e161c9-b3d1-4448-9cd9-403a0b6ffa3a-maas-api-openshift-ca\") pod \"82e161c9-b3d1-4448-9cd9-403a0b6ffa3a\" (UID: \"82e161c9-b3d1-4448-9cd9-403a0b6ffa3a\") "
Oct 30 20:59:33.601185 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:33.601149    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82e161c9-b3d1-4448-9cd9-403a0b6ffa3a-maas-api-openshift-ca" pod "82e161c9-b3d1-4448-9cd9-403a0b6ffa3a" (UID: "82e161c9-b3d1-4448-9cd9-403a0b6ffa3a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:59:33.602784 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:33.602756    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82e161c9-b3d1-4448-9cd9-403a0b6ffa3a-maas-api-xks-ca" pod "82e161c9-b3d1-4448-9cd9-403a0b6ffa3a" (UID: "82e161c9-b3d1-4448-9cd9-403a0b6ffa3a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:59:33.603442 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:33.603405    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82e161c9-b3d1-4448-9cd9-403a0b6ffa3a-kube-api-access-8gb5j" pod "82e161c9-b3d1-4448-9cd9-403a0b6ffa3a" (UID: "82e161c9-b3d1-4448-9cd9-403a0b6ffa3a"). InnerVolumeSpecName "kube-api-access-8gb5j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:59:33.701952 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:33.701919    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dplnl\" (UniqueName: \"kubernetes.io/projected/680a80b6-b2fd-4aa5-9128-23193a693221-kube-api-access-dplnl\") pod \"maas-api-revoke-sub-d92a4e2c-mvb6w\" (UID: \"680a80b6-b2fd-4aa5-9128-23193a693221\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d92a4e2c-mvb6w"
Oct 30 20:59:33.702109 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:33.702085    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/680a80b6-b2fd-4aa5-9128-23193a693221-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d92a4e2c-mvb6w\" (UID: \"680a80b6-b2fd-4aa5-9128-23193a693221\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d92a4e2c-mvb6w"
Oct 30 20:59:33.702170 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:33.702135    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/680a80b6-b2fd-4aa5-9128-23193a693221-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d92a4e2c-mvb6w\" (UID: \"680a80b6-b2fd-4aa5-9128-23193a693221\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d92a4e2c-mvb6w"
Oct 30 20:59:33.702244 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:33.702230    4521 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82e161c9-b3d1-4448-9cd9-403a0b6ffa3a-maas-api-xks-ca\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:59:33.702277 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:33.702250    4521 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8gb5j\" (UniqueName: \"kubernetes.io/projected/82e161c9-b3d1-4448-9cd9-403a0b6ffa3a-kube-api-access-8gb5j\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:59:33.702277 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:33.702261    4521 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82e161c9-b3d1-4448-9cd9-403a0b6ffa3a-maas-api-openshift-ca\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:59:33.803062 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:33.803013    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/680a80b6-b2fd-4aa5-9128-23193a693221-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d92a4e2c-mvb6w\" (UID: \"680a80b6-b2fd-4aa5-9128-23193a693221\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d92a4e2c-mvb6w"
Oct 30 20:59:33.803242 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:33.803076    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/680a80b6-b2fd-4aa5-9128-23193a693221-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d92a4e2c-mvb6w\" (UID: \"680a80b6-b2fd-4aa5-9128-23193a693221\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d92a4e2c-mvb6w"
Oct 30 20:59:33.803242 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:33.803113    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dplnl\" (UniqueName: \"kubernetes.io/projected/680a80b6-b2fd-4aa5-9128-23193a693221-kube-api-access-dplnl\") pod \"maas-api-revoke-sub-d92a4e2c-mvb6w\" (UID: \"680a80b6-b2fd-4aa5-9128-23193a693221\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d92a4e2c-mvb6w"
Oct 30 20:59:33.804019 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:33.803996    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/680a80b6-b2fd-4aa5-9128-23193a693221-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d92a4e2c-mvb6w\" (UID: \"680a80b6-b2fd-4aa5-9128-23193a693221\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d92a4e2c-mvb6w"
Oct 30 20:59:33.805549 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:33.805528    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/680a80b6-b2fd-4aa5-9128-23193a693221-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d92a4e2c-mvb6w\" (UID: \"680a80b6-b2fd-4aa5-9128-23193a693221\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d92a4e2c-mvb6w"
Oct 30 20:59:33.816735 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:33.816704    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dplnl\" (UniqueName: \"kubernetes.io/projected/680a80b6-b2fd-4aa5-9128-23193a693221-kube-api-access-dplnl\") pod \"maas-api-revoke-sub-d92a4e2c-mvb6w\" (UID: \"680a80b6-b2fd-4aa5-9128-23193a693221\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d92a4e2c-mvb6w"
Oct 30 20:59:33.884232 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:33.884145    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d92a4e2c-mvb6w"
Oct 30 20:59:34.165195 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:34.165155    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d92a4e2c-mvb6w"]
Oct 30 20:59:34.165663 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:59:34.165636    4521 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod680a80b6_b2fd_4aa5_9128_23193a693221.slice/crio-cb1759859a1c37a6b78bc166c9ade7a497d0c800960b121c3840b062484ecb58 WatchSource:0}: Error finding container cb1759859a1c37a6b78bc166c9ade7a497d0c800960b121c3840b062484ecb58: Status 404 returned error can't find the container with id cb1759859a1c37a6b78bc166c9ade7a497d0c800960b121c3840b062484ecb58
Oct 30 20:59:34.382302 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:34.382265    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf7689f5-sjgsr"
Oct 30 20:59:34.382503 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:34.382273    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf7689f5-sjgsr" event={"ID":"82e161c9-b3d1-4448-9cd9-403a0b6ffa3a","Type":"ContainerDied","Data":"10ae5f38786f7f0a60d46885c276c4a5365d3b507ecc443908747a93717693a4"}
Oct 30 20:59:34.382503 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:34.382378    4521 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="10ae5f38786f7f0a60d46885c276c4a5365d3b507ecc443908747a93717693a4"
Oct 30 20:59:34.383688 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:34.383663    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d92a4e2c-mvb6w" event={"ID":"680a80b6-b2fd-4aa5-9128-23193a693221","Type":"ContainerStarted","Data":"cb1759859a1c37a6b78bc166c9ade7a497d0c800960b121c3840b062484ecb58"}
Oct 30 20:59:34.385257 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:34.385233    4521 generic.go:356] "Generic (PLEG): container finished" podID="eecab7a4-8911-4b5f-b6fa-97727ae3e1cd" containerID="0418f0b50bc6be455544a63d9b0f7a5b46a38b2cfb0505813ba8049f9afc2596" exitCode=0
Oct 30 20:59:34.385405 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:34.385286    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11299c87-dlggc" event={"ID":"eecab7a4-8911-4b5f-b6fa-97727ae3e1cd","Type":"ContainerDied","Data":"0418f0b50bc6be455544a63d9b0f7a5b46a38b2cfb0505813ba8049f9afc2596"}
Oct 30 20:59:34.385405 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:34.385358    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11299c87-dlggc" event={"ID":"eecab7a4-8911-4b5f-b6fa-97727ae3e1cd","Type":"ContainerStarted","Data":"aa6ff878cbe28b13e0877ed5ff1d31020ba78aa2ce52faf2503ee0429b688ebd"}
Oct 30 20:59:35.393695 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:35.393575    4521 generic.go:356] "Generic (PLEG): container finished" podID="680a80b6-b2fd-4aa5-9128-23193a693221" containerID="59ce9780cf45903b387544fef5b2a1fe8b0b84f8f3b7e3ceb033000e23a67e83" exitCode=0
Oct 30 20:59:35.393695 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:35.393661    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d92a4e2c-mvb6w" event={"ID":"680a80b6-b2fd-4aa5-9128-23193a693221","Type":"ContainerDied","Data":"59ce9780cf45903b387544fef5b2a1fe8b0b84f8f3b7e3ceb033000e23a67e83"}
Oct 30 20:59:35.563164 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:35.563139    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11299c87-dlggc"
Oct 30 20:59:35.723161 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:35.723127    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eecab7a4-8911-4b5f-b6fa-97727ae3e1cd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eecab7a4-8911-4b5f-b6fa-97727ae3e1cd-maas-api-xks-ca\") pod \"eecab7a4-8911-4b5f-b6fa-97727ae3e1cd\" (UID: \"eecab7a4-8911-4b5f-b6fa-97727ae3e1cd\") "
Oct 30 20:59:35.723401 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:35.723188    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eecab7a4-8911-4b5f-b6fa-97727ae3e1cd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eecab7a4-8911-4b5f-b6fa-97727ae3e1cd-maas-api-openshift-ca\") pod \"eecab7a4-8911-4b5f-b6fa-97727ae3e1cd\" (UID: \"eecab7a4-8911-4b5f-b6fa-97727ae3e1cd\") "
Oct 30 20:59:35.723401 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:35.723227    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eecab7a4-8911-4b5f-b6fa-97727ae3e1cd-kube-api-access-z5jmx\" (UniqueName: \"kubernetes.io/projected/eecab7a4-8911-4b5f-b6fa-97727ae3e1cd-kube-api-access-z5jmx\") pod \"eecab7a4-8911-4b5f-b6fa-97727ae3e1cd\" (UID: \"eecab7a4-8911-4b5f-b6fa-97727ae3e1cd\") "
Oct 30 20:59:35.723692 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:35.723659    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eecab7a4-8911-4b5f-b6fa-97727ae3e1cd-maas-api-openshift-ca" pod "eecab7a4-8911-4b5f-b6fa-97727ae3e1cd" (UID: "eecab7a4-8911-4b5f-b6fa-97727ae3e1cd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:59:35.725430 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:35.725396    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eecab7a4-8911-4b5f-b6fa-97727ae3e1cd-maas-api-xks-ca" pod "eecab7a4-8911-4b5f-b6fa-97727ae3e1cd" (UID: "eecab7a4-8911-4b5f-b6fa-97727ae3e1cd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:59:35.725430 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:35.725419    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eecab7a4-8911-4b5f-b6fa-97727ae3e1cd-kube-api-access-z5jmx" pod "eecab7a4-8911-4b5f-b6fa-97727ae3e1cd" (UID: "eecab7a4-8911-4b5f-b6fa-97727ae3e1cd"). InnerVolumeSpecName "kube-api-access-z5jmx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:59:35.824306 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:35.824266    4521 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eecab7a4-8911-4b5f-b6fa-97727ae3e1cd-maas-api-xks-ca\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:59:35.824306 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:35.824305    4521 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eecab7a4-8911-4b5f-b6fa-97727ae3e1cd-maas-api-openshift-ca\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:59:35.824573 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:35.824348    4521 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5jmx\" (UniqueName: \"kubernetes.io/projected/eecab7a4-8911-4b5f-b6fa-97727ae3e1cd-kube-api-access-z5jmx\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:59:36.400584 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:36.400552    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11299c87-dlggc"
Oct 30 20:59:36.400986 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:36.400579    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11299c87-dlggc" event={"ID":"eecab7a4-8911-4b5f-b6fa-97727ae3e1cd","Type":"ContainerDied","Data":"aa6ff878cbe28b13e0877ed5ff1d31020ba78aa2ce52faf2503ee0429b688ebd"}
Oct 30 20:59:36.400986 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:36.400663    4521 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aa6ff878cbe28b13e0877ed5ff1d31020ba78aa2ce52faf2503ee0429b688ebd"
Oct 30 20:59:36.545907 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:36.545881    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d92a4e2c-mvb6w"
Oct 30 20:59:36.634301 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:36.634256    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/680a80b6-b2fd-4aa5-9128-23193a693221-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/680a80b6-b2fd-4aa5-9128-23193a693221-maas-api-openshift-ca\") pod \"680a80b6-b2fd-4aa5-9128-23193a693221\" (UID: \"680a80b6-b2fd-4aa5-9128-23193a693221\") "
Oct 30 20:59:36.634485 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:36.634368    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/680a80b6-b2fd-4aa5-9128-23193a693221-kube-api-access-dplnl\" (UniqueName: \"kubernetes.io/projected/680a80b6-b2fd-4aa5-9128-23193a693221-kube-api-access-dplnl\") pod \"680a80b6-b2fd-4aa5-9128-23193a693221\" (UID: \"680a80b6-b2fd-4aa5-9128-23193a693221\") "
Oct 30 20:59:36.634485 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:36.634477    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/680a80b6-b2fd-4aa5-9128-23193a693221-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/680a80b6-b2fd-4aa5-9128-23193a693221-maas-api-xks-ca\") pod \"680a80b6-b2fd-4aa5-9128-23193a693221\" (UID: \"680a80b6-b2fd-4aa5-9128-23193a693221\") "
Oct 30 20:59:36.634661 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:36.634599    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/680a80b6-b2fd-4aa5-9128-23193a693221-maas-api-openshift-ca" pod "680a80b6-b2fd-4aa5-9128-23193a693221" (UID: "680a80b6-b2fd-4aa5-9128-23193a693221"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:59:36.634887 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:36.634859    4521 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/680a80b6-b2fd-4aa5-9128-23193a693221-maas-api-openshift-ca\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:59:36.636877 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:36.636851    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/680a80b6-b2fd-4aa5-9128-23193a693221-maas-api-xks-ca" pod "680a80b6-b2fd-4aa5-9128-23193a693221" (UID: "680a80b6-b2fd-4aa5-9128-23193a693221"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:59:36.637046 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:36.637029    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/680a80b6-b2fd-4aa5-9128-23193a693221-kube-api-access-dplnl" pod "680a80b6-b2fd-4aa5-9128-23193a693221" (UID: "680a80b6-b2fd-4aa5-9128-23193a693221"). InnerVolumeSpecName "kube-api-access-dplnl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:59:36.736143 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:36.736099    4521 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dplnl\" (UniqueName: \"kubernetes.io/projected/680a80b6-b2fd-4aa5-9128-23193a693221-kube-api-access-dplnl\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:59:36.736143 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:36.736137    4521 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/680a80b6-b2fd-4aa5-9128-23193a693221-maas-api-xks-ca\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:59:37.407895 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:37.407863    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d92a4e2c-mvb6w"
Oct 30 20:59:37.407895 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:37.407891    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d92a4e2c-mvb6w" event={"ID":"680a80b6-b2fd-4aa5-9128-23193a693221","Type":"ContainerDied","Data":"cb1759859a1c37a6b78bc166c9ade7a497d0c800960b121c3840b062484ecb58"}
Oct 30 20:59:37.408373 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:37.407943    4521 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cb1759859a1c37a6b78bc166c9ade7a497d0c800960b121c3840b062484ecb58"
Oct 30 20:59:40.852484 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:40.852442    4521 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe8d59bc-chscd"]
Oct 30 20:59:40.853064 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:40.853046    4521 cpu_manager.go:414] "containerMap: removing container" podUID="eecab7a4-8911-4b5f-b6fa-97727ae3e1cd" containerName="revoke-keys"
Oct 30 20:59:40.853104 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:40.853068    4521 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eecab7a4-8911-4b5f-b6fa-97727ae3e1cd" containerName="revoke-keys"
Oct 30 20:59:40.853137 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:40.853109    4521 cpu_manager.go:414] "containerMap: removing container" podUID="680a80b6-b2fd-4aa5-9128-23193a693221" containerName="revoke-keys"
Oct 30 20:59:40.853137 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:40.853117    4521 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="680a80b6-b2fd-4aa5-9128-23193a693221" containerName="revoke-keys"
Oct 30 20:59:40.853241 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:40.853230    4521 memory_manager.go:365] "RemoveStaleState removing state" podUID="680a80b6-b2fd-4aa5-9128-23193a693221" containerName="revoke-keys"
Oct 30 20:59:40.853279 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:40.853248    4521 memory_manager.go:365] "RemoveStaleState removing state" podUID="eecab7a4-8911-4b5f-b6fa-97727ae3e1cd" containerName="revoke-keys"
Oct 30 20:59:40.857196 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:40.857174    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe8d59bc-chscd"
Oct 30 20:59:40.865968 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:40.865937    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe8d59bc-chscd"]
Oct 30 20:59:40.866102 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:40.865953    4521 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c45pw\""
Oct 30 20:59:40.866102 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:40.866009    4521 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:59:40.980968 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:40.980932    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/737f9761-4c67-41ca-9311-904bbfc50df4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe8d59bc-chscd\" (UID: \"737f9761-4c67-41ca-9311-904bbfc50df4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe8d59bc-chscd"
Oct 30 20:59:40.981134 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:40.981053    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcpz8\" (UniqueName: \"kubernetes.io/projected/737f9761-4c67-41ca-9311-904bbfc50df4-kube-api-access-qcpz8\") pod \"maas-api-revoke-sub-fe8d59bc-chscd\" (UID: \"737f9761-4c67-41ca-9311-904bbfc50df4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe8d59bc-chscd"
Oct 30 20:59:40.981227 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:40.981139    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/737f9761-4c67-41ca-9311-904bbfc50df4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe8d59bc-chscd\" (UID: \"737f9761-4c67-41ca-9311-904bbfc50df4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe8d59bc-chscd"
Oct 30 20:59:41.082372 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:41.082318    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/737f9761-4c67-41ca-9311-904bbfc50df4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe8d59bc-chscd\" (UID: \"737f9761-4c67-41ca-9311-904bbfc50df4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe8d59bc-chscd"
Oct 30 20:59:41.082527 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:41.082425    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcpz8\" (UniqueName: \"kubernetes.io/projected/737f9761-4c67-41ca-9311-904bbfc50df4-kube-api-access-qcpz8\") pod \"maas-api-revoke-sub-fe8d59bc-chscd\" (UID: \"737f9761-4c67-41ca-9311-904bbfc50df4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe8d59bc-chscd"
Oct 30 20:59:41.082527 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:41.082463    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/737f9761-4c67-41ca-9311-904bbfc50df4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe8d59bc-chscd\" (UID: \"737f9761-4c67-41ca-9311-904bbfc50df4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe8d59bc-chscd"
Oct 30 20:59:41.083243 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:41.083221    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/737f9761-4c67-41ca-9311-904bbfc50df4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe8d59bc-chscd\" (UID: \"737f9761-4c67-41ca-9311-904bbfc50df4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe8d59bc-chscd"
Oct 30 20:59:41.084688 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:41.084665    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/737f9761-4c67-41ca-9311-904bbfc50df4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe8d59bc-chscd\" (UID: \"737f9761-4c67-41ca-9311-904bbfc50df4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe8d59bc-chscd"
Oct 30 20:59:41.095115 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:41.095083    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcpz8\" (UniqueName: \"kubernetes.io/projected/737f9761-4c67-41ca-9311-904bbfc50df4-kube-api-access-qcpz8\") pod \"maas-api-revoke-sub-fe8d59bc-chscd\" (UID: \"737f9761-4c67-41ca-9311-904bbfc50df4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe8d59bc-chscd"
Oct 30 20:59:41.175181 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:41.175096    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe8d59bc-chscd"
Oct 30 20:59:41.455871 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:41.455843    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe8d59bc-chscd"]
Oct 30 20:59:41.457278 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:59:41.457247    4521 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod737f9761_4c67_41ca_9311_904bbfc50df4.slice/crio-1f57415afb09feda2156eb7d35b059e8b54fec9a415a88f7a02874fb5fccf487 WatchSource:0}: Error finding container 1f57415afb09feda2156eb7d35b059e8b54fec9a415a88f7a02874fb5fccf487: Status 404 returned error can't find the container with id 1f57415afb09feda2156eb7d35b059e8b54fec9a415a88f7a02874fb5fccf487
Oct 30 20:59:42.442524 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:42.442484    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe8d59bc-chscd" event={"ID":"737f9761-4c67-41ca-9311-904bbfc50df4","Type":"ContainerStarted","Data":"3fc4626481d2cecd58e9b2aae3fda4080331b32837a935da4cc89c8144b36353"}
Oct 30 20:59:42.443028 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:42.442550    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe8d59bc-chscd" event={"ID":"737f9761-4c67-41ca-9311-904bbfc50df4","Type":"ContainerStarted","Data":"1f57415afb09feda2156eb7d35b059e8b54fec9a415a88f7a02874fb5fccf487"}
Oct 30 20:59:42.463565 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:42.463508    4521 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe8d59bc-chscd" podStartSLOduration=2.463494008 podStartE2EDuration="2.463494008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:59:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:59:42.460763274 +0000 UTC m=+1249.806384513" watchObservedRunningTime="2026-10-30 20:59:42.463494008 +0000 UTC m=+1249.809115245"
Oct 30 20:59:43.451886 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:43.451850    4521 generic.go:356] "Generic (PLEG): container finished" podID="737f9761-4c67-41ca-9311-904bbfc50df4" containerID="3fc4626481d2cecd58e9b2aae3fda4080331b32837a935da4cc89c8144b36353" exitCode=0
Oct 30 20:59:43.452401 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:43.451916    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe8d59bc-chscd" event={"ID":"737f9761-4c67-41ca-9311-904bbfc50df4","Type":"ContainerDied","Data":"3fc4626481d2cecd58e9b2aae3fda4080331b32837a935da4cc89c8144b36353"}
Oct 30 20:59:44.245687 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:44.245643    4521 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd72df56-n6dtl"]
Oct 30 20:59:44.249361 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:44.249316    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd72df56-n6dtl"
Oct 30 20:59:44.268514 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:44.268488    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd72df56-n6dtl"]
Oct 30 20:59:44.328983 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:44.328951    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/769224a3-a1ff-458c-b43b-c802b6311ba4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dd72df56-n6dtl\" (UID: \"769224a3-a1ff-458c-b43b-c802b6311ba4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd72df56-n6dtl"
Oct 30 20:59:44.329137 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:44.329018    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9prs\" (UniqueName: \"kubernetes.io/projected/769224a3-a1ff-458c-b43b-c802b6311ba4-kube-api-access-r9prs\") pod \"maas-api-revoke-sub-dd72df56-n6dtl\" (UID: \"769224a3-a1ff-458c-b43b-c802b6311ba4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd72df56-n6dtl"
Oct 30 20:59:44.329180 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:44.329137    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/769224a3-a1ff-458c-b43b-c802b6311ba4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dd72df56-n6dtl\" (UID: \"769224a3-a1ff-458c-b43b-c802b6311ba4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd72df56-n6dtl"
Oct 30 20:59:44.430204 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:44.430167    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9prs\" (UniqueName: \"kubernetes.io/projected/769224a3-a1ff-458c-b43b-c802b6311ba4-kube-api-access-r9prs\") pod \"maas-api-revoke-sub-dd72df56-n6dtl\" (UID: \"769224a3-a1ff-458c-b43b-c802b6311ba4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd72df56-n6dtl"
Oct 30 20:59:44.430541 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:44.430214    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/769224a3-a1ff-458c-b43b-c802b6311ba4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dd72df56-n6dtl\" (UID: \"769224a3-a1ff-458c-b43b-c802b6311ba4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd72df56-n6dtl"
Oct 30 20:59:44.430541 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:44.430275    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/769224a3-a1ff-458c-b43b-c802b6311ba4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dd72df56-n6dtl\" (UID: \"769224a3-a1ff-458c-b43b-c802b6311ba4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd72df56-n6dtl"
Oct 30 20:59:44.431174 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:44.431147    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/769224a3-a1ff-458c-b43b-c802b6311ba4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dd72df56-n6dtl\" (UID: \"769224a3-a1ff-458c-b43b-c802b6311ba4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd72df56-n6dtl"
Oct 30 20:59:44.432753 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:44.432728    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/769224a3-a1ff-458c-b43b-c802b6311ba4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dd72df56-n6dtl\" (UID: \"769224a3-a1ff-458c-b43b-c802b6311ba4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd72df56-n6dtl"
Oct 30 20:59:44.443429 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:44.443397    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9prs\" (UniqueName: \"kubernetes.io/projected/769224a3-a1ff-458c-b43b-c802b6311ba4-kube-api-access-r9prs\") pod \"maas-api-revoke-sub-dd72df56-n6dtl\" (UID: \"769224a3-a1ff-458c-b43b-c802b6311ba4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd72df56-n6dtl"
Oct 30 20:59:44.563211 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:44.563176    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd72df56-n6dtl"
Oct 30 20:59:44.610896 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:44.610866    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe8d59bc-chscd"
Oct 30 20:59:44.642855 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:44.640746    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/737f9761-4c67-41ca-9311-904bbfc50df4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/737f9761-4c67-41ca-9311-904bbfc50df4-maas-api-xks-ca\") pod \"737f9761-4c67-41ca-9311-904bbfc50df4\" (UID: \"737f9761-4c67-41ca-9311-904bbfc50df4\") "
Oct 30 20:59:44.642855 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:44.640807    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/737f9761-4c67-41ca-9311-904bbfc50df4-kube-api-access-qcpz8\" (UniqueName: \"kubernetes.io/projected/737f9761-4c67-41ca-9311-904bbfc50df4-kube-api-access-qcpz8\") pod \"737f9761-4c67-41ca-9311-904bbfc50df4\" (UID: \"737f9761-4c67-41ca-9311-904bbfc50df4\") "
Oct 30 20:59:44.642855 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:44.640930    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/737f9761-4c67-41ca-9311-904bbfc50df4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/737f9761-4c67-41ca-9311-904bbfc50df4-maas-api-openshift-ca\") pod \"737f9761-4c67-41ca-9311-904bbfc50df4\" (UID: \"737f9761-4c67-41ca-9311-904bbfc50df4\") "
Oct 30 20:59:44.642855 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:44.642161    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/737f9761-4c67-41ca-9311-904bbfc50df4-maas-api-openshift-ca" pod "737f9761-4c67-41ca-9311-904bbfc50df4" (UID: "737f9761-4c67-41ca-9311-904bbfc50df4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:59:44.648358 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:44.647798    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/737f9761-4c67-41ca-9311-904bbfc50df4-kube-api-access-qcpz8" pod "737f9761-4c67-41ca-9311-904bbfc50df4" (UID: "737f9761-4c67-41ca-9311-904bbfc50df4"). InnerVolumeSpecName "kube-api-access-qcpz8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:59:44.648358 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:44.648057    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/737f9761-4c67-41ca-9311-904bbfc50df4-maas-api-xks-ca" pod "737f9761-4c67-41ca-9311-904bbfc50df4" (UID: "737f9761-4c67-41ca-9311-904bbfc50df4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:59:44.742202 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:44.742160    4521 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/737f9761-4c67-41ca-9311-904bbfc50df4-maas-api-openshift-ca\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:59:44.742202 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:44.742194    4521 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/737f9761-4c67-41ca-9311-904bbfc50df4-maas-api-xks-ca\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:59:44.742202 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:44.742208    4521 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qcpz8\" (UniqueName: \"kubernetes.io/projected/737f9761-4c67-41ca-9311-904bbfc50df4-kube-api-access-qcpz8\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:59:44.850709 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:44.850676    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd72df56-n6dtl"]
Oct 30 20:59:44.851419 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:59:44.851395    4521 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod769224a3_a1ff_458c_b43b_c802b6311ba4.slice/crio-6c7fe7eb8a7d2a5355551047e21006f9b076e736efbedb678245ed6134b70fa7 WatchSource:0}: Error finding container 6c7fe7eb8a7d2a5355551047e21006f9b076e736efbedb678245ed6134b70fa7: Status 404 returned error can't find the container with id 6c7fe7eb8a7d2a5355551047e21006f9b076e736efbedb678245ed6134b70fa7
Oct 30 20:59:45.465787 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:45.465757    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe8d59bc-chscd"
Oct 30 20:59:45.465965 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:45.465794    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe8d59bc-chscd" event={"ID":"737f9761-4c67-41ca-9311-904bbfc50df4","Type":"ContainerDied","Data":"1f57415afb09feda2156eb7d35b059e8b54fec9a415a88f7a02874fb5fccf487"}
Oct 30 20:59:45.465965 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:45.465843    4521 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1f57415afb09feda2156eb7d35b059e8b54fec9a415a88f7a02874fb5fccf487"
Oct 30 20:59:45.467559 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:45.467525    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd72df56-n6dtl" event={"ID":"769224a3-a1ff-458c-b43b-c802b6311ba4","Type":"ContainerStarted","Data":"c57dbe64ca675865d5af4b195ff48207773f7c234f161c2cd3954776a7ad6a88"}
Oct 30 20:59:45.467696 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:45.467567    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd72df56-n6dtl" event={"ID":"769224a3-a1ff-458c-b43b-c802b6311ba4","Type":"ContainerStarted","Data":"6c7fe7eb8a7d2a5355551047e21006f9b076e736efbedb678245ed6134b70fa7"}
Oct 30 20:59:45.486640 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:45.486603    4521 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd72df56-n6dtl" podStartSLOduration=1.486590526 podStartE2EDuration="1.486590526s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:59:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:59:45.485809146 +0000 UTC m=+1252.831430396" watchObservedRunningTime="2026-10-30 20:59:45.486590526 +0000 UTC m=+1252.832211763"
Oct 30 20:59:46.430638 ip-10-0-131-125 kubenswrapper[4521]: E1030 20:59:46.430589    4521 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lkn2k" containerName="extractor"
Oct 30 20:59:46.474910 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:46.474874    4521 generic.go:356] "Generic (PLEG): container finished" podID="769224a3-a1ff-458c-b43b-c802b6311ba4" containerID="c57dbe64ca675865d5af4b195ff48207773f7c234f161c2cd3954776a7ad6a88" exitCode=0
Oct 30 20:59:46.475084 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:46.474941    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd72df56-n6dtl" event={"ID":"769224a3-a1ff-458c-b43b-c802b6311ba4","Type":"ContainerDied","Data":"c57dbe64ca675865d5af4b195ff48207773f7c234f161c2cd3954776a7ad6a88"}
Oct 30 20:59:47.286716 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:47.286678    4521 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5a591e6-d2vtx"]
Oct 30 20:59:47.287311 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:47.287290    4521 cpu_manager.go:414] "containerMap: removing container" podUID="737f9761-4c67-41ca-9311-904bbfc50df4" containerName="revoke-keys"
Oct 30 20:59:47.287387 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:47.287312    4521 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="737f9761-4c67-41ca-9311-904bbfc50df4" containerName="revoke-keys"
Oct 30 20:59:47.287448 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:47.287436    4521 memory_manager.go:365] "RemoveStaleState removing state" podUID="737f9761-4c67-41ca-9311-904bbfc50df4" containerName="revoke-keys"
Oct 30 20:59:47.291012 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:47.290992    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5a591e6-d2vtx"
Oct 30 20:59:47.300198 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:47.300165    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5a591e6-d2vtx"]
Oct 30 20:59:47.373058 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:47.373024    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1016070-22d2-4278-8fe0-782cb3608dcf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b5a591e6-d2vtx\" (UID: \"c1016070-22d2-4278-8fe0-782cb3608dcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5a591e6-d2vtx"
Oct 30 20:59:47.373238 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:47.373074    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1016070-22d2-4278-8fe0-782cb3608dcf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b5a591e6-d2vtx\" (UID: \"c1016070-22d2-4278-8fe0-782cb3608dcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5a591e6-d2vtx"
Oct 30 20:59:47.373238 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:47.373105    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dtd7\" (UniqueName: \"kubernetes.io/projected/c1016070-22d2-4278-8fe0-782cb3608dcf-kube-api-access-2dtd7\") pod \"maas-api-revoke-sub-b5a591e6-d2vtx\" (UID: \"c1016070-22d2-4278-8fe0-782cb3608dcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5a591e6-d2vtx"
Oct 30 20:59:47.474447 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:47.474410    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1016070-22d2-4278-8fe0-782cb3608dcf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b5a591e6-d2vtx\" (UID: \"c1016070-22d2-4278-8fe0-782cb3608dcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5a591e6-d2vtx"
Oct 30 20:59:47.474834 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:47.474465    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1016070-22d2-4278-8fe0-782cb3608dcf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b5a591e6-d2vtx\" (UID: \"c1016070-22d2-4278-8fe0-782cb3608dcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5a591e6-d2vtx"
Oct 30 20:59:47.474834 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:47.474489    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dtd7\" (UniqueName: \"kubernetes.io/projected/c1016070-22d2-4278-8fe0-782cb3608dcf-kube-api-access-2dtd7\") pod \"maas-api-revoke-sub-b5a591e6-d2vtx\" (UID: \"c1016070-22d2-4278-8fe0-782cb3608dcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5a591e6-d2vtx"
Oct 30 20:59:47.475259 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:47.475230    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1016070-22d2-4278-8fe0-782cb3608dcf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b5a591e6-d2vtx\" (UID: \"c1016070-22d2-4278-8fe0-782cb3608dcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5a591e6-d2vtx"
Oct 30 20:59:47.476947 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:47.476928    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1016070-22d2-4278-8fe0-782cb3608dcf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b5a591e6-d2vtx\" (UID: \"c1016070-22d2-4278-8fe0-782cb3608dcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5a591e6-d2vtx"
Oct 30 20:59:47.490317 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:47.490284    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dtd7\" (UniqueName: \"kubernetes.io/projected/c1016070-22d2-4278-8fe0-782cb3608dcf-kube-api-access-2dtd7\") pod \"maas-api-revoke-sub-b5a591e6-d2vtx\" (UID: \"c1016070-22d2-4278-8fe0-782cb3608dcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5a591e6-d2vtx"
Oct 30 20:59:47.605813 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:47.605772    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5a591e6-d2vtx"
Oct 30 20:59:47.644494 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:47.644467    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd72df56-n6dtl"
Oct 30 20:59:47.677611 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:47.677341    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/769224a3-a1ff-458c-b43b-c802b6311ba4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/769224a3-a1ff-458c-b43b-c802b6311ba4-maas-api-openshift-ca\") pod \"769224a3-a1ff-458c-b43b-c802b6311ba4\" (UID: \"769224a3-a1ff-458c-b43b-c802b6311ba4\") "
Oct 30 20:59:47.677611 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:47.677423    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/769224a3-a1ff-458c-b43b-c802b6311ba4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/769224a3-a1ff-458c-b43b-c802b6311ba4-maas-api-xks-ca\") pod \"769224a3-a1ff-458c-b43b-c802b6311ba4\" (UID: \"769224a3-a1ff-458c-b43b-c802b6311ba4\") "
Oct 30 20:59:47.677611 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:47.677507    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/769224a3-a1ff-458c-b43b-c802b6311ba4-kube-api-access-r9prs\" (UniqueName: \"kubernetes.io/projected/769224a3-a1ff-458c-b43b-c802b6311ba4-kube-api-access-r9prs\") pod \"769224a3-a1ff-458c-b43b-c802b6311ba4\" (UID: \"769224a3-a1ff-458c-b43b-c802b6311ba4\") "
Oct 30 20:59:47.677890 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:47.677677    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/769224a3-a1ff-458c-b43b-c802b6311ba4-maas-api-openshift-ca" pod "769224a3-a1ff-458c-b43b-c802b6311ba4" (UID: "769224a3-a1ff-458c-b43b-c802b6311ba4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:59:47.677890 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:47.677840    4521 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/769224a3-a1ff-458c-b43b-c802b6311ba4-maas-api-openshift-ca\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:59:47.684581 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:47.684537    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/769224a3-a1ff-458c-b43b-c802b6311ba4-kube-api-access-r9prs" pod "769224a3-a1ff-458c-b43b-c802b6311ba4" (UID: "769224a3-a1ff-458c-b43b-c802b6311ba4"). InnerVolumeSpecName "kube-api-access-r9prs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:59:47.684900 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:47.684872    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/769224a3-a1ff-458c-b43b-c802b6311ba4-maas-api-xks-ca" pod "769224a3-a1ff-458c-b43b-c802b6311ba4" (UID: "769224a3-a1ff-458c-b43b-c802b6311ba4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:59:47.778728 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:47.778693    4521 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/769224a3-a1ff-458c-b43b-c802b6311ba4-maas-api-xks-ca\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:59:47.778728 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:47.778723    4521 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9prs\" (UniqueName: \"kubernetes.io/projected/769224a3-a1ff-458c-b43b-c802b6311ba4-kube-api-access-r9prs\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:59:47.886598 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:47.886570    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5a591e6-d2vtx"]
Oct 30 20:59:47.888357 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:59:47.888313    4521 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1016070_22d2_4278_8fe0_782cb3608dcf.slice/crio-644c1ecc15b59ff26246fdbc42898dc37f01d94e387822ae4d8fb826e08536b4 WatchSource:0}: Error finding container 644c1ecc15b59ff26246fdbc42898dc37f01d94e387822ae4d8fb826e08536b4: Status 404 returned error can't find the container with id 644c1ecc15b59ff26246fdbc42898dc37f01d94e387822ae4d8fb826e08536b4
Oct 30 20:59:48.489611 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:48.489514    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5a591e6-d2vtx" event={"ID":"c1016070-22d2-4278-8fe0-782cb3608dcf","Type":"ContainerStarted","Data":"d2a53a9d6acd94400a50aac4040bf09b7743f31404303a3a558c37d721f1d052"}
Oct 30 20:59:48.489611 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:48.489576    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5a591e6-d2vtx" event={"ID":"c1016070-22d2-4278-8fe0-782cb3608dcf","Type":"ContainerStarted","Data":"644c1ecc15b59ff26246fdbc42898dc37f01d94e387822ae4d8fb826e08536b4"}
Oct 30 20:59:48.490998 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:48.490980    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd72df56-n6dtl"
Oct 30 20:59:48.491098 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:48.490977    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd72df56-n6dtl" event={"ID":"769224a3-a1ff-458c-b43b-c802b6311ba4","Type":"ContainerDied","Data":"6c7fe7eb8a7d2a5355551047e21006f9b076e736efbedb678245ed6134b70fa7"}
Oct 30 20:59:48.491147 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:48.491117    4521 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6c7fe7eb8a7d2a5355551047e21006f9b076e736efbedb678245ed6134b70fa7"
Oct 30 20:59:48.511973 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:48.511912    4521 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5a591e6-d2vtx" podStartSLOduration=1.511895534 podStartE2EDuration="1.511895534s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:59:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:59:48.507415616 +0000 UTC m=+1255.853036879" watchObservedRunningTime="2026-10-30 20:59:48.511895534 +0000 UTC m=+1255.857516773"
Oct 30 20:59:49.176840 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:49.176761    4521 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3c3bec5-mv7m5"]
Oct 30 20:59:49.177191 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:49.177179    4521 cpu_manager.go:414] "containerMap: removing container" podUID="769224a3-a1ff-458c-b43b-c802b6311ba4" containerName="revoke-keys"
Oct 30 20:59:49.177229 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:49.177192    4521 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="769224a3-a1ff-458c-b43b-c802b6311ba4" containerName="revoke-keys"
Oct 30 20:59:49.177292 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:49.177283    4521 memory_manager.go:365] "RemoveStaleState removing state" podUID="769224a3-a1ff-458c-b43b-c802b6311ba4" containerName="revoke-keys"
Oct 30 20:59:49.180470 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:49.180455    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3c3bec5-mv7m5"
Oct 30 20:59:49.190173 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:49.190144    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3c3bec5-mv7m5"]
Oct 30 20:59:49.292718 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:49.292679    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06a6a3a4-f7a3-4235-a80b-8e2163968e35-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3c3bec5-mv7m5\" (UID: \"06a6a3a4-f7a3-4235-a80b-8e2163968e35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3c3bec5-mv7m5"
Oct 30 20:59:49.292917 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:49.292732    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dh4j7\" (UniqueName: \"kubernetes.io/projected/06a6a3a4-f7a3-4235-a80b-8e2163968e35-kube-api-access-dh4j7\") pod \"maas-api-revoke-sub-d3c3bec5-mv7m5\" (UID: \"06a6a3a4-f7a3-4235-a80b-8e2163968e35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3c3bec5-mv7m5"
Oct 30 20:59:49.292917 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:49.292820    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06a6a3a4-f7a3-4235-a80b-8e2163968e35-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3c3bec5-mv7m5\" (UID: \"06a6a3a4-f7a3-4235-a80b-8e2163968e35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3c3bec5-mv7m5"
Oct 30 20:59:49.393907 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:49.393865    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06a6a3a4-f7a3-4235-a80b-8e2163968e35-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3c3bec5-mv7m5\" (UID: \"06a6a3a4-f7a3-4235-a80b-8e2163968e35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3c3bec5-mv7m5"
Oct 30 20:59:49.393907 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:49.393911    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dh4j7\" (UniqueName: \"kubernetes.io/projected/06a6a3a4-f7a3-4235-a80b-8e2163968e35-kube-api-access-dh4j7\") pod \"maas-api-revoke-sub-d3c3bec5-mv7m5\" (UID: \"06a6a3a4-f7a3-4235-a80b-8e2163968e35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3c3bec5-mv7m5"
Oct 30 20:59:49.394220 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:49.394029    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06a6a3a4-f7a3-4235-a80b-8e2163968e35-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3c3bec5-mv7m5\" (UID: \"06a6a3a4-f7a3-4235-a80b-8e2163968e35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3c3bec5-mv7m5"
Oct 30 20:59:49.394642 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:49.394624    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06a6a3a4-f7a3-4235-a80b-8e2163968e35-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3c3bec5-mv7m5\" (UID: \"06a6a3a4-f7a3-4235-a80b-8e2163968e35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3c3bec5-mv7m5"
Oct 30 20:59:49.396209 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:49.396187    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06a6a3a4-f7a3-4235-a80b-8e2163968e35-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3c3bec5-mv7m5\" (UID: \"06a6a3a4-f7a3-4235-a80b-8e2163968e35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3c3bec5-mv7m5"
Oct 30 20:59:49.406468 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:49.406443    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dh4j7\" (UniqueName: \"kubernetes.io/projected/06a6a3a4-f7a3-4235-a80b-8e2163968e35-kube-api-access-dh4j7\") pod \"maas-api-revoke-sub-d3c3bec5-mv7m5\" (UID: \"06a6a3a4-f7a3-4235-a80b-8e2163968e35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3c3bec5-mv7m5"
Oct 30 20:59:49.495349 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:49.495293    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3c3bec5-mv7m5"
Oct 30 20:59:49.499028 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:49.498996    4521 generic.go:356] "Generic (PLEG): container finished" podID="c1016070-22d2-4278-8fe0-782cb3608dcf" containerID="d2a53a9d6acd94400a50aac4040bf09b7743f31404303a3a558c37d721f1d052" exitCode=0
Oct 30 20:59:49.499166 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:49.499093    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5a591e6-d2vtx" event={"ID":"c1016070-22d2-4278-8fe0-782cb3608dcf","Type":"ContainerDied","Data":"d2a53a9d6acd94400a50aac4040bf09b7743f31404303a3a558c37d721f1d052"}
Oct 30 20:59:49.782371 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:49.782338    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3c3bec5-mv7m5"]
Oct 30 20:59:49.783388 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:59:49.783354    4521 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06a6a3a4_f7a3_4235_a80b_8e2163968e35.slice/crio-ce4ca1a4c77f47c0084486eb25aafbc7acba257fb44dbb7b7e0f30862fcf5e50 WatchSource:0}: Error finding container ce4ca1a4c77f47c0084486eb25aafbc7acba257fb44dbb7b7e0f30862fcf5e50: Status 404 returned error can't find the container with id ce4ca1a4c77f47c0084486eb25aafbc7acba257fb44dbb7b7e0f30862fcf5e50
Oct 30 20:59:50.506202 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:50.506158    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3c3bec5-mv7m5" event={"ID":"06a6a3a4-f7a3-4235-a80b-8e2163968e35","Type":"ContainerStarted","Data":"d5a2583ce54c0167f86336c241e0152bfbe1b1ad07d8eae2e2b4aee8bc397c66"}
Oct 30 20:59:50.506706 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:50.506225    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3c3bec5-mv7m5" event={"ID":"06a6a3a4-f7a3-4235-a80b-8e2163968e35","Type":"ContainerStarted","Data":"ce4ca1a4c77f47c0084486eb25aafbc7acba257fb44dbb7b7e0f30862fcf5e50"}
Oct 30 20:59:50.528585 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:50.528528    4521 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3c3bec5-mv7m5" podStartSLOduration=1.528512485 podStartE2EDuration="1.528512485s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:59:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:59:50.526130083 +0000 UTC m=+1257.871751326" watchObservedRunningTime="2026-10-30 20:59:50.528512485 +0000 UTC m=+1257.874133722"
Oct 30 20:59:50.595243 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:50.595184    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4kxb"]
Oct 30 20:59:50.734724 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:50.734702    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5a591e6-d2vtx"
Oct 30 20:59:50.808669 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:50.808600    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1016070-22d2-4278-8fe0-782cb3608dcf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1016070-22d2-4278-8fe0-782cb3608dcf-maas-api-xks-ca\") pod \"c1016070-22d2-4278-8fe0-782cb3608dcf\" (UID: \"c1016070-22d2-4278-8fe0-782cb3608dcf\") "
Oct 30 20:59:50.808669 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:50.808658    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1016070-22d2-4278-8fe0-782cb3608dcf-kube-api-access-2dtd7\" (UniqueName: \"kubernetes.io/projected/c1016070-22d2-4278-8fe0-782cb3608dcf-kube-api-access-2dtd7\") pod \"c1016070-22d2-4278-8fe0-782cb3608dcf\" (UID: \"c1016070-22d2-4278-8fe0-782cb3608dcf\") "
Oct 30 20:59:50.808848 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:50.808796    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1016070-22d2-4278-8fe0-782cb3608dcf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1016070-22d2-4278-8fe0-782cb3608dcf-maas-api-openshift-ca\") pod \"c1016070-22d2-4278-8fe0-782cb3608dcf\" (UID: \"c1016070-22d2-4278-8fe0-782cb3608dcf\") "
Oct 30 20:59:50.809199 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:50.809167    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1016070-22d2-4278-8fe0-782cb3608dcf-maas-api-openshift-ca" pod "c1016070-22d2-4278-8fe0-782cb3608dcf" (UID: "c1016070-22d2-4278-8fe0-782cb3608dcf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:59:50.810827 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:50.810799    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1016070-22d2-4278-8fe0-782cb3608dcf-kube-api-access-2dtd7" pod "c1016070-22d2-4278-8fe0-782cb3608dcf" (UID: "c1016070-22d2-4278-8fe0-782cb3608dcf"). InnerVolumeSpecName "kube-api-access-2dtd7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:59:50.810954 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:50.810930    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1016070-22d2-4278-8fe0-782cb3608dcf-maas-api-xks-ca" pod "c1016070-22d2-4278-8fe0-782cb3608dcf" (UID: "c1016070-22d2-4278-8fe0-782cb3608dcf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:59:50.910300 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:50.910266    4521 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1016070-22d2-4278-8fe0-782cb3608dcf-maas-api-openshift-ca\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:59:50.910300 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:50.910298    4521 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1016070-22d2-4278-8fe0-782cb3608dcf-maas-api-xks-ca\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:59:50.910300 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:50.910308    4521 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dtd7\" (UniqueName: \"kubernetes.io/projected/c1016070-22d2-4278-8fe0-782cb3608dcf-kube-api-access-2dtd7\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:59:51.513565 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:51.513533    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5a591e6-d2vtx"
Oct 30 20:59:51.514002 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:51.513567    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5a591e6-d2vtx" event={"ID":"c1016070-22d2-4278-8fe0-782cb3608dcf","Type":"ContainerDied","Data":"644c1ecc15b59ff26246fdbc42898dc37f01d94e387822ae4d8fb826e08536b4"}
Oct 30 20:59:51.514002 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:51.513620    4521 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="644c1ecc15b59ff26246fdbc42898dc37f01d94e387822ae4d8fb826e08536b4"
Oct 30 20:59:51.515314 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:51.515284    4521 generic.go:356] "Generic (PLEG): container finished" podID="06a6a3a4-f7a3-4235-a80b-8e2163968e35" containerID="d5a2583ce54c0167f86336c241e0152bfbe1b1ad07d8eae2e2b4aee8bc397c66" exitCode=0
Oct 30 20:59:51.515454 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:51.515363    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3c3bec5-mv7m5" event={"ID":"06a6a3a4-f7a3-4235-a80b-8e2163968e35","Type":"ContainerDied","Data":"d5a2583ce54c0167f86336c241e0152bfbe1b1ad07d8eae2e2b4aee8bc397c66"}
Oct 30 20:59:52.973749 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:52.973464    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3c3bec5-mv7m5"
Oct 30 20:59:53.056762 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:53.056726    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06a6a3a4-f7a3-4235-a80b-8e2163968e35-kube-api-access-dh4j7\" (UniqueName: \"kubernetes.io/projected/06a6a3a4-f7a3-4235-a80b-8e2163968e35-kube-api-access-dh4j7\") pod \"06a6a3a4-f7a3-4235-a80b-8e2163968e35\" (UID: \"06a6a3a4-f7a3-4235-a80b-8e2163968e35\") "
Oct 30 20:59:53.057033 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:53.057012    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06a6a3a4-f7a3-4235-a80b-8e2163968e35-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06a6a3a4-f7a3-4235-a80b-8e2163968e35-maas-api-openshift-ca\") pod \"06a6a3a4-f7a3-4235-a80b-8e2163968e35\" (UID: \"06a6a3a4-f7a3-4235-a80b-8e2163968e35\") "
Oct 30 20:59:53.057160 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:53.057144    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/06a6a3a4-f7a3-4235-a80b-8e2163968e35-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06a6a3a4-f7a3-4235-a80b-8e2163968e35-maas-api-xks-ca\") pod \"06a6a3a4-f7a3-4235-a80b-8e2163968e35\" (UID: \"06a6a3a4-f7a3-4235-a80b-8e2163968e35\") "
Oct 30 20:59:53.061580 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:53.061552    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06a6a3a4-f7a3-4235-a80b-8e2163968e35-maas-api-openshift-ca" pod "06a6a3a4-f7a3-4235-a80b-8e2163968e35" (UID: "06a6a3a4-f7a3-4235-a80b-8e2163968e35"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:59:53.089231 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:53.089174    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06a6a3a4-f7a3-4235-a80b-8e2163968e35-kube-api-access-dh4j7" pod "06a6a3a4-f7a3-4235-a80b-8e2163968e35" (UID: "06a6a3a4-f7a3-4235-a80b-8e2163968e35"). InnerVolumeSpecName "kube-api-access-dh4j7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:59:53.120380 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:53.119402    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06a6a3a4-f7a3-4235-a80b-8e2163968e35-maas-api-xks-ca" pod "06a6a3a4-f7a3-4235-a80b-8e2163968e35" (UID: "06a6a3a4-f7a3-4235-a80b-8e2163968e35"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:59:53.161381 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:53.161028    4521 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dh4j7\" (UniqueName: \"kubernetes.io/projected/06a6a3a4-f7a3-4235-a80b-8e2163968e35-kube-api-access-dh4j7\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:59:53.161381 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:53.161066    4521 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06a6a3a4-f7a3-4235-a80b-8e2163968e35-maas-api-openshift-ca\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:59:53.161381 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:53.161082    4521 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06a6a3a4-f7a3-4235-a80b-8e2163968e35-maas-api-xks-ca\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:59:53.550346 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:53.547151    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3c3bec5-mv7m5" event={"ID":"06a6a3a4-f7a3-4235-a80b-8e2163968e35","Type":"ContainerDied","Data":"ce4ca1a4c77f47c0084486eb25aafbc7acba257fb44dbb7b7e0f30862fcf5e50"}
Oct 30 20:59:53.550346 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:53.547217    4521 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ce4ca1a4c77f47c0084486eb25aafbc7acba257fb44dbb7b7e0f30862fcf5e50"
Oct 30 20:59:53.550346 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:53.547291    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3c3bec5-mv7m5"
Oct 30 20:59:55.396555 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:55.396516    4521 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-hlv6g"]
Oct 30 20:59:55.396959 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:55.396944    4521 cpu_manager.go:414] "containerMap: removing container" podUID="06a6a3a4-f7a3-4235-a80b-8e2163968e35" containerName="revoke-keys"
Oct 30 20:59:55.397015 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:55.396959    4521 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06a6a3a4-f7a3-4235-a80b-8e2163968e35" containerName="revoke-keys"
Oct 30 20:59:55.397015 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:55.396982    4521 cpu_manager.go:414] "containerMap: removing container" podUID="c1016070-22d2-4278-8fe0-782cb3608dcf" containerName="revoke-keys"
Oct 30 20:59:55.397015 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:55.396987    4521 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1016070-22d2-4278-8fe0-782cb3608dcf" containerName="revoke-keys"
Oct 30 20:59:55.397124 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:55.397068    4521 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1016070-22d2-4278-8fe0-782cb3608dcf" containerName="revoke-keys"
Oct 30 20:59:55.397124 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:55.397080    4521 memory_manager.go:365] "RemoveStaleState removing state" podUID="06a6a3a4-f7a3-4235-a80b-8e2163968e35" containerName="revoke-keys"
Oct 30 20:59:55.404841 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:55.404818    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-hlv6g"
Oct 30 20:59:55.409754 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:55.409732    4521 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-4a0c5b-openshift-default-dockercfg-ngx6w\""
Oct 30 20:59:55.416976 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:55.416947    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-hlv6g"]
Oct 30 20:59:55.504011 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:55.503977    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f860c372-4493-4c25-b46a-a6d3af710390-istio-podinfo\") pod \"e2e-worker-w0-4a0c5b-openshift-default-676865df99-hlv6g\" (UID: \"f860c372-4493-4c25-b46a-a6d3af710390\") " pod="openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-hlv6g"
Oct 30 20:59:55.504227 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:55.504033    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f860c372-4493-4c25-b46a-a6d3af710390-workload-socket\") pod \"e2e-worker-w0-4a0c5b-openshift-default-676865df99-hlv6g\" (UID: \"f860c372-4493-4c25-b46a-a6d3af710390\") " pod="openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-hlv6g"
Oct 30 20:59:55.504227 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:55.504166    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f860c372-4493-4c25-b46a-a6d3af710390-istio-data\") pod \"e2e-worker-w0-4a0c5b-openshift-default-676865df99-hlv6g\" (UID: \"f860c372-4493-4c25-b46a-a6d3af710390\") " pod="openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-hlv6g"
Oct 30 20:59:55.504227 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:55.504219    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f860c372-4493-4c25-b46a-a6d3af710390-istio-envoy\") pod \"e2e-worker-w0-4a0c5b-openshift-default-676865df99-hlv6g\" (UID: \"f860c372-4493-4c25-b46a-a6d3af710390\") " pod="openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-hlv6g"
Oct 30 20:59:55.504408 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:55.504244    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kp9px\" (UniqueName: \"kubernetes.io/projected/f860c372-4493-4c25-b46a-a6d3af710390-kube-api-access-kp9px\") pod \"e2e-worker-w0-4a0c5b-openshift-default-676865df99-hlv6g\" (UID: \"f860c372-4493-4c25-b46a-a6d3af710390\") " pod="openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-hlv6g"
Oct 30 20:59:55.504408 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:55.504303    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f860c372-4493-4c25-b46a-a6d3af710390-credential-socket\") pod \"e2e-worker-w0-4a0c5b-openshift-default-676865df99-hlv6g\" (UID: \"f860c372-4493-4c25-b46a-a6d3af710390\") " pod="openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-hlv6g"
Oct 30 20:59:55.504408 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:55.504347    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f860c372-4493-4c25-b46a-a6d3af710390-workload-certs\") pod \"e2e-worker-w0-4a0c5b-openshift-default-676865df99-hlv6g\" (UID: \"f860c372-4493-4c25-b46a-a6d3af710390\") " pod="openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-hlv6g"
Oct 30 20:59:55.504551 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:55.504412    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f860c372-4493-4c25-b46a-a6d3af710390-istio-token\") pod \"e2e-worker-w0-4a0c5b-openshift-default-676865df99-hlv6g\" (UID: \"f860c372-4493-4c25-b46a-a6d3af710390\") " pod="openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-hlv6g"
Oct 30 20:59:55.504551 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:55.504442    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f860c372-4493-4c25-b46a-a6d3af710390-istiod-ca-cert\") pod \"e2e-worker-w0-4a0c5b-openshift-default-676865df99-hlv6g\" (UID: \"f860c372-4493-4c25-b46a-a6d3af710390\") " pod="openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-hlv6g"
Oct 30 20:59:55.604995 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:55.604958    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f860c372-4493-4c25-b46a-a6d3af710390-istio-podinfo\") pod \"e2e-worker-w0-4a0c5b-openshift-default-676865df99-hlv6g\" (UID: \"f860c372-4493-4c25-b46a-a6d3af710390\") " pod="openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-hlv6g"
Oct 30 20:59:55.605199 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:55.605017    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f860c372-4493-4c25-b46a-a6d3af710390-workload-socket\") pod \"e2e-worker-w0-4a0c5b-openshift-default-676865df99-hlv6g\" (UID: \"f860c372-4493-4c25-b46a-a6d3af710390\") " pod="openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-hlv6g"
Oct 30 20:59:55.605199 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:55.605064    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f860c372-4493-4c25-b46a-a6d3af710390-istio-data\") pod \"e2e-worker-w0-4a0c5b-openshift-default-676865df99-hlv6g\" (UID: \"f860c372-4493-4c25-b46a-a6d3af710390\") " pod="openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-hlv6g"
Oct 30 20:59:55.605199 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:55.605084    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f860c372-4493-4c25-b46a-a6d3af710390-istio-envoy\") pod \"e2e-worker-w0-4a0c5b-openshift-default-676865df99-hlv6g\" (UID: \"f860c372-4493-4c25-b46a-a6d3af710390\") " pod="openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-hlv6g"
Oct 30 20:59:55.605199 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:55.605116    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kp9px\" (UniqueName: \"kubernetes.io/projected/f860c372-4493-4c25-b46a-a6d3af710390-kube-api-access-kp9px\") pod \"e2e-worker-w0-4a0c5b-openshift-default-676865df99-hlv6g\" (UID: \"f860c372-4493-4c25-b46a-a6d3af710390\") " pod="openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-hlv6g"
Oct 30 20:59:55.605199 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:55.605160    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f860c372-4493-4c25-b46a-a6d3af710390-credential-socket\") pod \"e2e-worker-w0-4a0c5b-openshift-default-676865df99-hlv6g\" (UID: \"f860c372-4493-4c25-b46a-a6d3af710390\") " pod="openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-hlv6g"
Oct 30 20:59:55.605199 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:55.605187    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f860c372-4493-4c25-b46a-a6d3af710390-workload-certs\") pod \"e2e-worker-w0-4a0c5b-openshift-default-676865df99-hlv6g\" (UID: \"f860c372-4493-4c25-b46a-a6d3af710390\") " pod="openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-hlv6g"
Oct 30 20:59:55.605537 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:55.605222    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f860c372-4493-4c25-b46a-a6d3af710390-istio-token\") pod \"e2e-worker-w0-4a0c5b-openshift-default-676865df99-hlv6g\" (UID: \"f860c372-4493-4c25-b46a-a6d3af710390\") " pod="openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-hlv6g"
Oct 30 20:59:55.605537 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:55.605245    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f860c372-4493-4c25-b46a-a6d3af710390-istiod-ca-cert\") pod \"e2e-worker-w0-4a0c5b-openshift-default-676865df99-hlv6g\" (UID: \"f860c372-4493-4c25-b46a-a6d3af710390\") " pod="openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-hlv6g"
Oct 30 20:59:55.605641 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:55.605562    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f860c372-4493-4c25-b46a-a6d3af710390-workload-socket\") pod \"e2e-worker-w0-4a0c5b-openshift-default-676865df99-hlv6g\" (UID: \"f860c372-4493-4c25-b46a-a6d3af710390\") " pod="openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-hlv6g"
Oct 30 20:59:55.605641 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:55.605612    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f860c372-4493-4c25-b46a-a6d3af710390-istio-data\") pod \"e2e-worker-w0-4a0c5b-openshift-default-676865df99-hlv6g\" (UID: \"f860c372-4493-4c25-b46a-a6d3af710390\") " pod="openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-hlv6g"
Oct 30 20:59:55.605913 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:55.605878    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f860c372-4493-4c25-b46a-a6d3af710390-workload-certs\") pod \"e2e-worker-w0-4a0c5b-openshift-default-676865df99-hlv6g\" (UID: \"f860c372-4493-4c25-b46a-a6d3af710390\") " pod="openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-hlv6g"
Oct 30 20:59:55.605913 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:55.605900    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f860c372-4493-4c25-b46a-a6d3af710390-credential-socket\") pod \"e2e-worker-w0-4a0c5b-openshift-default-676865df99-hlv6g\" (UID: \"f860c372-4493-4c25-b46a-a6d3af710390\") " pod="openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-hlv6g"
Oct 30 20:59:55.606296 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:55.606274    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f860c372-4493-4c25-b46a-a6d3af710390-istiod-ca-cert\") pod \"e2e-worker-w0-4a0c5b-openshift-default-676865df99-hlv6g\" (UID: \"f860c372-4493-4c25-b46a-a6d3af710390\") " pod="openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-hlv6g"
Oct 30 20:59:55.607488 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:55.607469    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f860c372-4493-4c25-b46a-a6d3af710390-istio-envoy\") pod \"e2e-worker-w0-4a0c5b-openshift-default-676865df99-hlv6g\" (UID: \"f860c372-4493-4c25-b46a-a6d3af710390\") " pod="openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-hlv6g"
Oct 30 20:59:55.607644 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:55.607627    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f860c372-4493-4c25-b46a-a6d3af710390-istio-podinfo\") pod \"e2e-worker-w0-4a0c5b-openshift-default-676865df99-hlv6g\" (UID: \"f860c372-4493-4c25-b46a-a6d3af710390\") " pod="openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-hlv6g"
Oct 30 20:59:55.618571 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:55.618541    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kp9px\" (UniqueName: \"kubernetes.io/projected/f860c372-4493-4c25-b46a-a6d3af710390-kube-api-access-kp9px\") pod \"e2e-worker-w0-4a0c5b-openshift-default-676865df99-hlv6g\" (UID: \"f860c372-4493-4c25-b46a-a6d3af710390\") " pod="openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-hlv6g"
Oct 30 20:59:55.618718 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:55.618698    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f860c372-4493-4c25-b46a-a6d3af710390-istio-token\") pod \"e2e-worker-w0-4a0c5b-openshift-default-676865df99-hlv6g\" (UID: \"f860c372-4493-4c25-b46a-a6d3af710390\") " pod="openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-hlv6g"
Oct 30 20:59:55.735433 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:55.735396    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-hlv6g"
Oct 30 20:59:56.026714 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:59:56.026671    4521 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf860c372_4493_4c25_b46a_a6d3af710390.slice/crio-d08a5ea3aae2a3215a9800e6f9200b7bb7f399b060db228a11c439e59390bd63 WatchSource:0}: Error finding container d08a5ea3aae2a3215a9800e6f9200b7bb7f399b060db228a11c439e59390bd63: Status 404 returned error can't find the container with id d08a5ea3aae2a3215a9800e6f9200b7bb7f399b060db228a11c439e59390bd63
Oct 30 20:59:56.026973 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:56.026945    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-hlv6g"]
Oct 30 20:59:56.029569 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:56.029530    4521 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:59:56.029676 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:56.029613    4521 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:59:56.029676 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:56.029657    4521 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:59:56.574529 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:56.574491    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-hlv6g" event={"ID":"f860c372-4493-4c25-b46a-a6d3af710390","Type":"ContainerStarted","Data":"21a3de3b202c4f10c9229e21a63d3e7b1b879d9996e96d4490513f2692dea48a"}
Oct 30 20:59:56.574940 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:56.574553    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-hlv6g" event={"ID":"f860c372-4493-4c25-b46a-a6d3af710390","Type":"ContainerStarted","Data":"d08a5ea3aae2a3215a9800e6f9200b7bb7f399b060db228a11c439e59390bd63"}
Oct 30 20:59:56.604196 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:56.604132    4521 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-hlv6g" podStartSLOduration=1.604112073 podStartE2EDuration="1.604112073s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:59:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:59:56.601100271 +0000 UTC m=+1263.946721546" watchObservedRunningTime="2026-10-30 20:59:56.604112073 +0000 UTC m=+1263.949733293"
Oct 30 20:59:56.735692 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:56.735610    4521 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-hlv6g"
Oct 30 20:59:57.698643 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:57.698604    4521 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e488030-ngvh7"]
Oct 30 20:59:57.702562 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:57.702541    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e488030-ngvh7"
Oct 30 20:59:57.708861 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:57.708836    4521 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c45pw\""
Oct 30 20:59:57.709011 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:57.708878    4521 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:59:57.746350 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:57.745547    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e488030-ngvh7"]
Oct 30 20:59:57.747879 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:57.747846    4521 patch_prober.go:29] interesting pod/e2e-worker-w0-4a0c5b-openshift-default-676865df99-hlv6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Oct 30 20:59:57.748069 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:57.747902    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-hlv6g" podUID="f860c372-4493-4c25-b46a-a6d3af710390" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Oct 30 20:59:57.749062 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:57.749034    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3665eb7-48ee-4eef-a8c9-06e5ebfec690-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e488030-ngvh7\" (UID: \"d3665eb7-48ee-4eef-a8c9-06e5ebfec690\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e488030-ngvh7"
Oct 30 20:59:57.749169 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:57.749083    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3665eb7-48ee-4eef-a8c9-06e5ebfec690-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e488030-ngvh7\" (UID: \"d3665eb7-48ee-4eef-a8c9-06e5ebfec690\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e488030-ngvh7"
Oct 30 20:59:57.749169 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:57.749154    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56nbz\" (UniqueName: \"kubernetes.io/projected/d3665eb7-48ee-4eef-a8c9-06e5ebfec690-kube-api-access-56nbz\") pod \"maas-api-revoke-sub-2e488030-ngvh7\" (UID: \"d3665eb7-48ee-4eef-a8c9-06e5ebfec690\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e488030-ngvh7"
Oct 30 20:59:57.858745 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:57.857111    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3665eb7-48ee-4eef-a8c9-06e5ebfec690-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e488030-ngvh7\" (UID: \"d3665eb7-48ee-4eef-a8c9-06e5ebfec690\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e488030-ngvh7"
Oct 30 20:59:57.858745 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:57.857283    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3665eb7-48ee-4eef-a8c9-06e5ebfec690-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e488030-ngvh7\" (UID: \"d3665eb7-48ee-4eef-a8c9-06e5ebfec690\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e488030-ngvh7"
Oct 30 20:59:57.858745 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:57.857442    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56nbz\" (UniqueName: \"kubernetes.io/projected/d3665eb7-48ee-4eef-a8c9-06e5ebfec690-kube-api-access-56nbz\") pod \"maas-api-revoke-sub-2e488030-ngvh7\" (UID: \"d3665eb7-48ee-4eef-a8c9-06e5ebfec690\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e488030-ngvh7"
Oct 30 20:59:57.859629 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:57.859601    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3665eb7-48ee-4eef-a8c9-06e5ebfec690-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e488030-ngvh7\" (UID: \"d3665eb7-48ee-4eef-a8c9-06e5ebfec690\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e488030-ngvh7"
Oct 30 20:59:57.865682 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:57.865653    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3665eb7-48ee-4eef-a8c9-06e5ebfec690-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e488030-ngvh7\" (UID: \"d3665eb7-48ee-4eef-a8c9-06e5ebfec690\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e488030-ngvh7"
Oct 30 20:59:57.869891 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:57.869863    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56nbz\" (UniqueName: \"kubernetes.io/projected/d3665eb7-48ee-4eef-a8c9-06e5ebfec690-kube-api-access-56nbz\") pod \"maas-api-revoke-sub-2e488030-ngvh7\" (UID: \"d3665eb7-48ee-4eef-a8c9-06e5ebfec690\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e488030-ngvh7"
Oct 30 20:59:58.055393 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:58.055285    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e488030-ngvh7"
Oct 30 20:59:58.359255 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:58.359226    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e488030-ngvh7"]
Oct 30 20:59:58.359988 ip-10-0-131-125 kubenswrapper[4521]: W1030 20:59:58.359961    4521 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3665eb7_48ee_4eef_a8c9_06e5ebfec690.slice/crio-bb62927ced71f290ca7931326a3138e53afbbd77266c68d48d1d6e6f29865b78 WatchSource:0}: Error finding container bb62927ced71f290ca7931326a3138e53afbbd77266c68d48d1d6e6f29865b78: Status 404 returned error can't find the container with id bb62927ced71f290ca7931326a3138e53afbbd77266c68d48d1d6e6f29865b78
Oct 30 20:59:58.588738 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:58.588695    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e488030-ngvh7" event={"ID":"d3665eb7-48ee-4eef-a8c9-06e5ebfec690","Type":"ContainerStarted","Data":"bb62927ced71f290ca7931326a3138e53afbbd77266c68d48d1d6e6f29865b78"}
Oct 30 20:59:58.745529 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:58.745481    4521 patch_prober.go:29] interesting pod/e2e-worker-w0-4a0c5b-openshift-default-676865df99-hlv6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Oct 30 20:59:58.746108 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:58.745549    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-hlv6g" podUID="f860c372-4493-4c25-b46a-a6d3af710390" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Oct 30 20:59:59.596578 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:59.596492    4521 generic.go:356] "Generic (PLEG): container finished" podID="d3665eb7-48ee-4eef-a8c9-06e5ebfec690" containerID="0998c212c96ef56ae77040422eb0f141d6c4a819f3488164f3639521a2317857" exitCode=0
Oct 30 20:59:59.596758 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:59.596656    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e488030-ngvh7" event={"ID":"d3665eb7-48ee-4eef-a8c9-06e5ebfec690","Type":"ContainerDied","Data":"0998c212c96ef56ae77040422eb0f141d6c4a819f3488164f3639521a2317857"}
Oct 30 20:59:59.738914 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:59.738796    4521 patch_prober.go:29] interesting pod/e2e-worker-w0-4a0c5b-openshift-default-676865df99-hlv6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Oct 30 20:59:59.738914 ip-10-0-131-125 kubenswrapper[4521]: I1030 20:59:59.738880    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-hlv6g" podUID="f860c372-4493-4c25-b46a-a6d3af710390" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Oct 30 21:00:00.150990 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:00.150941    4521 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-65968b-29889900-hkfkj"]
Oct 30 21:00:00.155224 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:00.155201    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-65968b-29889900-hkfkj"
Oct 30 21:00:00.174145 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:00.173583    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-65968b-29889900-hkfkj"]
Oct 30 21:00:00.176889 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:00.176863    4521 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c4b379-29889900-sbvxl"]
Oct 30 21:00:00.190363 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:00.183975    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mjmq\" (UniqueName: \"kubernetes.io/projected/f0bc9c01-1d10-4e3d-9876-f24637b968ef-kube-api-access-7mjmq\") pod \"maas-api-key-cleanup-e2e-worker-w5-65968b-29889900-hkfkj\" (UID: \"f0bc9c01-1d10-4e3d-9876-f24637b968ef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-65968b-29889900-hkfkj"
Oct 30 21:00:00.190363 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:00.184059    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0bc9c01-1d10-4e3d-9876-f24637b968ef-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-65968b-29889900-hkfkj\" (UID: \"f0bc9c01-1d10-4e3d-9876-f24637b968ef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-65968b-29889900-hkfkj"
Oct 30 21:00:00.190363 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:00.184318    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c4b379-29889900-sbvxl"
Oct 30 21:00:00.190363 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:00.184896    4521 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-39fb96-29889900-sbp6w"]
Oct 30 21:00:00.197333 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:00.194267    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-39fb96-29889900-sbp6w"
Oct 30 21:00:00.200483 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:00.197936    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c4b379-29889900-sbvxl"]
Oct 30 21:00:00.206643 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:00.204978    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-39fb96-29889900-sbp6w"]
Oct 30 21:00:00.285397 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:00.285317    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8q4h\" (UniqueName: \"kubernetes.io/projected/49b3ef08-eca0-4a62-885d-bcf8fc23cb6d-kube-api-access-w8q4h\") pod \"maas-api-key-cleanup-e2e-worker-w6-39fb96-29889900-sbp6w\" (UID: \"49b3ef08-eca0-4a62-885d-bcf8fc23cb6d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-39fb96-29889900-sbp6w"
Oct 30 21:00:00.285397 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:00.285388    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7w2zc\" (UniqueName: \"kubernetes.io/projected/1e319e8b-6373-4ad3-b549-086f4a0c0f25-kube-api-access-7w2zc\") pod \"maas-api-key-cleanup-e2e-worker-w1-c4b379-29889900-sbvxl\" (UID: \"1e319e8b-6373-4ad3-b549-086f4a0c0f25\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c4b379-29889900-sbvxl"
Oct 30 21:00:00.285630 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:00.285414    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mjmq\" (UniqueName: \"kubernetes.io/projected/f0bc9c01-1d10-4e3d-9876-f24637b968ef-kube-api-access-7mjmq\") pod \"maas-api-key-cleanup-e2e-worker-w5-65968b-29889900-hkfkj\" (UID: \"f0bc9c01-1d10-4e3d-9876-f24637b968ef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-65968b-29889900-hkfkj"
Oct 30 21:00:00.285836 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:00.285821    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0bc9c01-1d10-4e3d-9876-f24637b968ef-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-65968b-29889900-hkfkj\" (UID: \"f0bc9c01-1d10-4e3d-9876-f24637b968ef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-65968b-29889900-hkfkj"
Oct 30 21:00:00.285909 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:00.285851    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/49b3ef08-eca0-4a62-885d-bcf8fc23cb6d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-39fb96-29889900-sbp6w\" (UID: \"49b3ef08-eca0-4a62-885d-bcf8fc23cb6d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-39fb96-29889900-sbp6w"
Oct 30 21:00:00.286050 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:00.286026    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e319e8b-6373-4ad3-b549-086f4a0c0f25-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-c4b379-29889900-sbvxl\" (UID: \"1e319e8b-6373-4ad3-b549-086f4a0c0f25\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c4b379-29889900-sbvxl"
Oct 30 21:00:00.286594 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:00.286572    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0bc9c01-1d10-4e3d-9876-f24637b968ef-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-65968b-29889900-hkfkj\" (UID: \"f0bc9c01-1d10-4e3d-9876-f24637b968ef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-65968b-29889900-hkfkj"
Oct 30 21:00:00.298400 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:00.298364    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mjmq\" (UniqueName: \"kubernetes.io/projected/f0bc9c01-1d10-4e3d-9876-f24637b968ef-kube-api-access-7mjmq\") pod \"maas-api-key-cleanup-e2e-worker-w5-65968b-29889900-hkfkj\" (UID: \"f0bc9c01-1d10-4e3d-9876-f24637b968ef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-65968b-29889900-hkfkj"
Oct 30 21:00:00.387132 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:00.387092    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e319e8b-6373-4ad3-b549-086f4a0c0f25-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-c4b379-29889900-sbvxl\" (UID: \"1e319e8b-6373-4ad3-b549-086f4a0c0f25\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c4b379-29889900-sbvxl"
Oct 30 21:00:00.387304 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:00.387182    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8q4h\" (UniqueName: \"kubernetes.io/projected/49b3ef08-eca0-4a62-885d-bcf8fc23cb6d-kube-api-access-w8q4h\") pod \"maas-api-key-cleanup-e2e-worker-w6-39fb96-29889900-sbp6w\" (UID: \"49b3ef08-eca0-4a62-885d-bcf8fc23cb6d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-39fb96-29889900-sbp6w"
Oct 30 21:00:00.387304 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:00.387209    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7w2zc\" (UniqueName: \"kubernetes.io/projected/1e319e8b-6373-4ad3-b549-086f4a0c0f25-kube-api-access-7w2zc\") pod \"maas-api-key-cleanup-e2e-worker-w1-c4b379-29889900-sbvxl\" (UID: \"1e319e8b-6373-4ad3-b549-086f4a0c0f25\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c4b379-29889900-sbvxl"
Oct 30 21:00:00.387304 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:00.387264    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/49b3ef08-eca0-4a62-885d-bcf8fc23cb6d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-39fb96-29889900-sbp6w\" (UID: \"49b3ef08-eca0-4a62-885d-bcf8fc23cb6d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-39fb96-29889900-sbp6w"
Oct 30 21:00:00.387914 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:00.387895    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/49b3ef08-eca0-4a62-885d-bcf8fc23cb6d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-39fb96-29889900-sbp6w\" (UID: \"49b3ef08-eca0-4a62-885d-bcf8fc23cb6d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-39fb96-29889900-sbp6w"
Oct 30 21:00:00.388335 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:00.388311    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e319e8b-6373-4ad3-b549-086f4a0c0f25-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-c4b379-29889900-sbvxl\" (UID: \"1e319e8b-6373-4ad3-b549-086f4a0c0f25\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c4b379-29889900-sbvxl"
Oct 30 21:00:00.402567 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:00.401156    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7w2zc\" (UniqueName: \"kubernetes.io/projected/1e319e8b-6373-4ad3-b549-086f4a0c0f25-kube-api-access-7w2zc\") pod \"maas-api-key-cleanup-e2e-worker-w1-c4b379-29889900-sbvxl\" (UID: \"1e319e8b-6373-4ad3-b549-086f4a0c0f25\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c4b379-29889900-sbvxl"
Oct 30 21:00:00.403181 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:00.403141    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8q4h\" (UniqueName: \"kubernetes.io/projected/49b3ef08-eca0-4a62-885d-bcf8fc23cb6d-kube-api-access-w8q4h\") pod \"maas-api-key-cleanup-e2e-worker-w6-39fb96-29889900-sbp6w\" (UID: \"49b3ef08-eca0-4a62-885d-bcf8fc23cb6d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-39fb96-29889900-sbp6w"
Oct 30 21:00:00.477056 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:00.476554    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-65968b-29889900-hkfkj"
Oct 30 21:00:00.536822 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:00.536566    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c4b379-29889900-sbvxl"
Oct 30 21:00:00.561071 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:00.561039    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-39fb96-29889900-sbp6w"
Oct 30 21:00:00.747351 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:00.739796    4521 patch_prober.go:29] interesting pod/e2e-worker-w0-4a0c5b-openshift-default-676865df99-hlv6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Oct 30 21:00:00.747351 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:00.739857    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-hlv6g" podUID="f860c372-4493-4c25-b46a-a6d3af710390" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Oct 30 21:00:00.847826 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:00.847630    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-65968b-29889900-hkfkj"]
Oct 30 21:00:00.849652 ip-10-0-131-125 kubenswrapper[4521]: W1030 21:00:00.849602    4521 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0bc9c01_1d10_4e3d_9876_f24637b968ef.slice/crio-9d3099faa50f306b41d109ef0a334938792a335f053270604c2e25b0cdf32c82 WatchSource:0}: Error finding container 9d3099faa50f306b41d109ef0a334938792a335f053270604c2e25b0cdf32c82: Status 404 returned error can't find the container with id 9d3099faa50f306b41d109ef0a334938792a335f053270604c2e25b0cdf32c82
Oct 30 21:00:00.899641 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:00.899614    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c4b379-29889900-sbvxl"]
Oct 30 21:00:00.900235 ip-10-0-131-125 kubenswrapper[4521]: W1030 21:00:00.900200    4521 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e319e8b_6373_4ad3_b549_086f4a0c0f25.slice/crio-2f326416916de7c833ef6b0775486e7e4c9b6ad4e47ca66d3a05eee890e13a1b WatchSource:0}: Error finding container 2f326416916de7c833ef6b0775486e7e4c9b6ad4e47ca66d3a05eee890e13a1b: Status 404 returned error can't find the container with id 2f326416916de7c833ef6b0775486e7e4c9b6ad4e47ca66d3a05eee890e13a1b
Oct 30 21:00:00.927860 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:00.909494    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-39fb96-29889900-sbp6w"]
Oct 30 21:00:01.068390 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:01.068360    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e488030-ngvh7"
Oct 30 21:00:01.098408 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:01.098379    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3665eb7-48ee-4eef-a8c9-06e5ebfec690-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3665eb7-48ee-4eef-a8c9-06e5ebfec690-maas-api-openshift-ca\") pod \"d3665eb7-48ee-4eef-a8c9-06e5ebfec690\" (UID: \"d3665eb7-48ee-4eef-a8c9-06e5ebfec690\") "
Oct 30 21:00:01.098559 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:01.098418    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3665eb7-48ee-4eef-a8c9-06e5ebfec690-kube-api-access-56nbz\" (UniqueName: \"kubernetes.io/projected/d3665eb7-48ee-4eef-a8c9-06e5ebfec690-kube-api-access-56nbz\") pod \"d3665eb7-48ee-4eef-a8c9-06e5ebfec690\" (UID: \"d3665eb7-48ee-4eef-a8c9-06e5ebfec690\") "
Oct 30 21:00:01.098559 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:01.098462    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3665eb7-48ee-4eef-a8c9-06e5ebfec690-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3665eb7-48ee-4eef-a8c9-06e5ebfec690-maas-api-xks-ca\") pod \"d3665eb7-48ee-4eef-a8c9-06e5ebfec690\" (UID: \"d3665eb7-48ee-4eef-a8c9-06e5ebfec690\") "
Oct 30 21:00:01.099664 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:01.099631    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3665eb7-48ee-4eef-a8c9-06e5ebfec690-maas-api-openshift-ca" pod "d3665eb7-48ee-4eef-a8c9-06e5ebfec690" (UID: "d3665eb7-48ee-4eef-a8c9-06e5ebfec690"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:00:01.101572 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:01.101539    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3665eb7-48ee-4eef-a8c9-06e5ebfec690-maas-api-xks-ca" pod "d3665eb7-48ee-4eef-a8c9-06e5ebfec690" (UID: "d3665eb7-48ee-4eef-a8c9-06e5ebfec690"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:00:01.102162 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:01.102140    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3665eb7-48ee-4eef-a8c9-06e5ebfec690-kube-api-access-56nbz" pod "d3665eb7-48ee-4eef-a8c9-06e5ebfec690" (UID: "d3665eb7-48ee-4eef-a8c9-06e5ebfec690"). InnerVolumeSpecName "kube-api-access-56nbz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:00:01.200134 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:01.200099    4521 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3665eb7-48ee-4eef-a8c9-06e5ebfec690-maas-api-openshift-ca\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:00:01.200134 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:01.200129    4521 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56nbz\" (UniqueName: \"kubernetes.io/projected/d3665eb7-48ee-4eef-a8c9-06e5ebfec690-kube-api-access-56nbz\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:00:01.200585 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:01.200141    4521 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3665eb7-48ee-4eef-a8c9-06e5ebfec690-maas-api-xks-ca\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:00:01.664134 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:01.664093    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-39fb96-29889900-sbp6w" event={"ID":"49b3ef08-eca0-4a62-885d-bcf8fc23cb6d","Type":"ContainerStarted","Data":"65e45032e661f3e59b0d6afef615908a9cdc09b939bfd47e9d8ae273af7f5043"}
Oct 30 21:00:01.664288 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:01.664194    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-39fb96-29889900-sbp6w" event={"ID":"49b3ef08-eca0-4a62-885d-bcf8fc23cb6d","Type":"ContainerStarted","Data":"51f30029c986e264fe8726611967e3523e8e3d720fcd25b6fa3e6e76b92dd5c9"}
Oct 30 21:00:01.665099 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:01.665068    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e488030-ngvh7" event={"ID":"d3665eb7-48ee-4eef-a8c9-06e5ebfec690","Type":"ContainerDied","Data":"bb62927ced71f290ca7931326a3138e53afbbd77266c68d48d1d6e6f29865b78"}
Oct 30 21:00:01.665250 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:01.665118    4521 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bb62927ced71f290ca7931326a3138e53afbbd77266c68d48d1d6e6f29865b78"
Oct 30 21:00:01.665250 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:01.665093    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e488030-ngvh7"
Oct 30 21:00:01.667241 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:01.666865    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-65968b-29889900-hkfkj" event={"ID":"f0bc9c01-1d10-4e3d-9876-f24637b968ef","Type":"ContainerStarted","Data":"ef8964114fe1841fddea74799dfc5b265df85802ad9ea372d915ca67ecbb1e35"}
Oct 30 21:00:01.667241 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:01.666900    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-65968b-29889900-hkfkj" event={"ID":"f0bc9c01-1d10-4e3d-9876-f24637b968ef","Type":"ContainerStarted","Data":"9d3099faa50f306b41d109ef0a334938792a335f053270604c2e25b0cdf32c82"}
Oct 30 21:00:01.669421 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:01.669354    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c4b379-29889900-sbvxl" event={"ID":"1e319e8b-6373-4ad3-b549-086f4a0c0f25","Type":"ContainerStarted","Data":"6357f80f48a77164ea54261b2a461394c6e0338290f840ee7aaa8d67c2539412"}
Oct 30 21:00:01.669421 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:01.669395    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c4b379-29889900-sbvxl" event={"ID":"1e319e8b-6373-4ad3-b549-086f4a0c0f25","Type":"ContainerStarted","Data":"2f326416916de7c833ef6b0775486e7e4c9b6ad4e47ca66d3a05eee890e13a1b"}
Oct 30 21:00:01.683834 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:01.683790    4521 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-39fb96-29889900-sbp6w" podStartSLOduration=1.6837778239999999 podStartE2EDuration="1.683777824s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:00:01.682494706 +0000 UTC m=+1269.028115944" watchObservedRunningTime="2026-10-30 21:00:01.683777824 +0000 UTC m=+1269.029399061"
Oct 30 21:00:01.706392 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:01.706333    4521 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c4b379-29889900-sbvxl" podStartSLOduration=1.706300784 podStartE2EDuration="1.706300784s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:00:01.703059972 +0000 UTC m=+1269.048681190" watchObservedRunningTime="2026-10-30 21:00:01.706300784 +0000 UTC m=+1269.051922049"
Oct 30 21:00:01.743349 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:01.742452    4521 patch_prober.go:29] interesting pod/e2e-worker-w0-4a0c5b-openshift-default-676865df99-hlv6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Oct 30 21:00:01.743349 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:01.742558    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-hlv6g" podUID="f860c372-4493-4c25-b46a-a6d3af710390" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Oct 30 21:00:01.745803 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:01.745746    4521 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-65968b-29889900-hkfkj" podStartSLOduration=1.745725371 podStartE2EDuration="1.745725371s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:00:01.730662844 +0000 UTC m=+1269.076284075" watchObservedRunningTime="2026-10-30 21:00:01.745725371 +0000 UTC m=+1269.091346610"
Oct 30 21:00:02.680347 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:02.680094    4521 generic.go:356] "Generic (PLEG): container finished" podID="49b3ef08-eca0-4a62-885d-bcf8fc23cb6d" containerID="65e45032e661f3e59b0d6afef615908a9cdc09b939bfd47e9d8ae273af7f5043" exitCode=60
Oct 30 21:00:02.680851 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:02.680391    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-39fb96-29889900-sbp6w" event={"ID":"49b3ef08-eca0-4a62-885d-bcf8fc23cb6d","Type":"ContainerDied","Data":"65e45032e661f3e59b0d6afef615908a9cdc09b939bfd47e9d8ae273af7f5043"}
Oct 30 21:00:02.680851 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:02.680824    4521 scope.go:122] "RemoveContainer" containerID="65e45032e661f3e59b0d6afef615908a9cdc09b939bfd47e9d8ae273af7f5043"
Oct 30 21:00:02.683762 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:02.683702    4521 generic.go:356] "Generic (PLEG): container finished" podID="f0bc9c01-1d10-4e3d-9876-f24637b968ef" containerID="ef8964114fe1841fddea74799dfc5b265df85802ad9ea372d915ca67ecbb1e35" exitCode=60
Oct 30 21:00:02.683915 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:02.683792    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-65968b-29889900-hkfkj" event={"ID":"f0bc9c01-1d10-4e3d-9876-f24637b968ef","Type":"ContainerDied","Data":"ef8964114fe1841fddea74799dfc5b265df85802ad9ea372d915ca67ecbb1e35"}
Oct 30 21:00:02.684131 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:02.684113    4521 scope.go:122] "RemoveContainer" containerID="ef8964114fe1841fddea74799dfc5b265df85802ad9ea372d915ca67ecbb1e35"
Oct 30 21:00:02.688416 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:02.688391    4521 generic.go:356] "Generic (PLEG): container finished" podID="1e319e8b-6373-4ad3-b549-086f4a0c0f25" containerID="6357f80f48a77164ea54261b2a461394c6e0338290f840ee7aaa8d67c2539412" exitCode=60
Oct 30 21:00:02.688511 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:02.688449    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c4b379-29889900-sbvxl" event={"ID":"1e319e8b-6373-4ad3-b549-086f4a0c0f25","Type":"ContainerDied","Data":"6357f80f48a77164ea54261b2a461394c6e0338290f840ee7aaa8d67c2539412"}
Oct 30 21:00:02.688736 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:02.688714    4521 scope.go:122] "RemoveContainer" containerID="6357f80f48a77164ea54261b2a461394c6e0338290f840ee7aaa8d67c2539412"
Oct 30 21:00:02.765404 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:02.764446    4521 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-hlv6g"
Oct 30 21:00:02.765404 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:02.765357    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-hlv6g"
Oct 30 21:00:02.766212 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:02.766187    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-hlv6g"
Oct 30 21:00:03.697142 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:03.697100    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-65968b-29889900-hkfkj" event={"ID":"f0bc9c01-1d10-4e3d-9876-f24637b968ef","Type":"ContainerStarted","Data":"b42a19e8e609608a0cf9a112877cc76091fe4dcada28ce1fb6cc83b85f77222c"}
Oct 30 21:00:03.699473 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:03.699449    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c4b379-29889900-sbvxl" event={"ID":"1e319e8b-6373-4ad3-b549-086f4a0c0f25","Type":"ContainerStarted","Data":"8354941073a53fcc812e0428dd49e48225c40acf3f7f36e0aefca53a4060fbbe"}
Oct 30 21:00:03.702724 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:03.702703    4521 generic.go:356] "Generic (PLEG): container finished" podID="49b3ef08-eca0-4a62-885d-bcf8fc23cb6d" containerID="7be1b30dd65665678dbc107591759f5cd9a166ad0356e7fae892af4552330fb3" exitCode=60
Oct 30 21:00:03.702818 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:03.702773    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-39fb96-29889900-sbp6w" event={"ID":"49b3ef08-eca0-4a62-885d-bcf8fc23cb6d","Type":"ContainerDied","Data":"7be1b30dd65665678dbc107591759f5cd9a166ad0356e7fae892af4552330fb3"}
Oct 30 21:00:03.702883 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:03.702825    4521 scope.go:122] "RemoveContainer" containerID="65e45032e661f3e59b0d6afef615908a9cdc09b939bfd47e9d8ae273af7f5043"
Oct 30 21:00:03.703128 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:03.703100    4521 scope.go:122] "RemoveContainer" containerID="7be1b30dd65665678dbc107591759f5cd9a166ad0356e7fae892af4552330fb3"
Oct 30 21:00:03.703440 ip-10-0-131-125 kubenswrapper[4521]: E1030 21:00:03.703415    4521 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-39fb96-29889900-sbp6w_odh-ai-gateway-infra(49b3ef08-eca0-4a62-885d-bcf8fc23cb6d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-39fb96-29889900-sbp6w" podUID="49b3ef08-eca0-4a62-885d-bcf8fc23cb6d"
Oct 30 21:00:03.771362 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:03.769553    4521 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-5a8f94/distinct-w3-5a8f94-kserve-5f8b544d59-mf95j"]
Oct 30 21:00:03.771362 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:03.770230    4521 cpu_manager.go:414] "containerMap: removing container" podUID="d3665eb7-48ee-4eef-a8c9-06e5ebfec690" containerName="revoke-keys"
Oct 30 21:00:03.771362 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:03.770251    4521 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3665eb7-48ee-4eef-a8c9-06e5ebfec690" containerName="revoke-keys"
Oct 30 21:00:03.771362 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:03.770448    4521 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3665eb7-48ee-4eef-a8c9-06e5ebfec690" containerName="revoke-keys"
Oct 30 21:00:03.774873 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:03.774846    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-5a8f94/distinct-w3-5a8f94-kserve-5f8b544d59-mf95j"
Oct 30 21:00:03.782161 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:03.781870    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-5a8f94/distinct-w3-5a8f94-kserve-5f8b544d59-mf95j"]
Oct 30 21:00:03.789798 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:03.788753    4521 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-5a8f94\"/\"distinct-w3-5a8f94-kserve-self-signed-certs\""
Oct 30 21:00:03.832904 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:03.832872    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bstjz\" (UniqueName: \"kubernetes.io/projected/b8ac1cca-4ebd-474c-b32f-a3595a4cd847-kube-api-access-bstjz\") pod \"distinct-w3-5a8f94-kserve-5f8b544d59-mf95j\" (UID: \"b8ac1cca-4ebd-474c-b32f-a3595a4cd847\") " pod="e2e-models-e2e-worker-w3-5a8f94/distinct-w3-5a8f94-kserve-5f8b544d59-mf95j"
Oct 30 21:00:03.833691 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:03.833667    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8ac1cca-4ebd-474c-b32f-a3595a4cd847-home\") pod \"distinct-w3-5a8f94-kserve-5f8b544d59-mf95j\" (UID: \"b8ac1cca-4ebd-474c-b32f-a3595a4cd847\") " pod="e2e-models-e2e-worker-w3-5a8f94/distinct-w3-5a8f94-kserve-5f8b544d59-mf95j"
Oct 30 21:00:03.833904 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:03.833888    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8ac1cca-4ebd-474c-b32f-a3595a4cd847-tls-certs\") pod \"distinct-w3-5a8f94-kserve-5f8b544d59-mf95j\" (UID: \"b8ac1cca-4ebd-474c-b32f-a3595a4cd847\") " pod="e2e-models-e2e-worker-w3-5a8f94/distinct-w3-5a8f94-kserve-5f8b544d59-mf95j"
Oct 30 21:00:03.835824 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:03.835804    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8ac1cca-4ebd-474c-b32f-a3595a4cd847-dshm\") pod \"distinct-w3-5a8f94-kserve-5f8b544d59-mf95j\" (UID: \"b8ac1cca-4ebd-474c-b32f-a3595a4cd847\") " pod="e2e-models-e2e-worker-w3-5a8f94/distinct-w3-5a8f94-kserve-5f8b544d59-mf95j"
Oct 30 21:00:03.836021 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:03.836005    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8ac1cca-4ebd-474c-b32f-a3595a4cd847-model-cache\") pod \"distinct-w3-5a8f94-kserve-5f8b544d59-mf95j\" (UID: \"b8ac1cca-4ebd-474c-b32f-a3595a4cd847\") " pod="e2e-models-e2e-worker-w3-5a8f94/distinct-w3-5a8f94-kserve-5f8b544d59-mf95j"
Oct 30 21:00:03.836441 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:03.836424    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8ac1cca-4ebd-474c-b32f-a3595a4cd847-tmp-dir\") pod \"distinct-w3-5a8f94-kserve-5f8b544d59-mf95j\" (UID: \"b8ac1cca-4ebd-474c-b32f-a3595a4cd847\") " pod="e2e-models-e2e-worker-w3-5a8f94/distinct-w3-5a8f94-kserve-5f8b544d59-mf95j"
Oct 30 21:00:03.938456 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:03.938418    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bstjz\" (UniqueName: \"kubernetes.io/projected/b8ac1cca-4ebd-474c-b32f-a3595a4cd847-kube-api-access-bstjz\") pod \"distinct-w3-5a8f94-kserve-5f8b544d59-mf95j\" (UID: \"b8ac1cca-4ebd-474c-b32f-a3595a4cd847\") " pod="e2e-models-e2e-worker-w3-5a8f94/distinct-w3-5a8f94-kserve-5f8b544d59-mf95j"
Oct 30 21:00:03.938600 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:03.938504    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8ac1cca-4ebd-474c-b32f-a3595a4cd847-home\") pod \"distinct-w3-5a8f94-kserve-5f8b544d59-mf95j\" (UID: \"b8ac1cca-4ebd-474c-b32f-a3595a4cd847\") " pod="e2e-models-e2e-worker-w3-5a8f94/distinct-w3-5a8f94-kserve-5f8b544d59-mf95j"
Oct 30 21:00:03.938600 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:03.938541    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8ac1cca-4ebd-474c-b32f-a3595a4cd847-tls-certs\") pod \"distinct-w3-5a8f94-kserve-5f8b544d59-mf95j\" (UID: \"b8ac1cca-4ebd-474c-b32f-a3595a4cd847\") " pod="e2e-models-e2e-worker-w3-5a8f94/distinct-w3-5a8f94-kserve-5f8b544d59-mf95j"
Oct 30 21:00:03.938600 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:03.938579    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8ac1cca-4ebd-474c-b32f-a3595a4cd847-dshm\") pod \"distinct-w3-5a8f94-kserve-5f8b544d59-mf95j\" (UID: \"b8ac1cca-4ebd-474c-b32f-a3595a4cd847\") " pod="e2e-models-e2e-worker-w3-5a8f94/distinct-w3-5a8f94-kserve-5f8b544d59-mf95j"
Oct 30 21:00:03.938773 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:03.938621    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8ac1cca-4ebd-474c-b32f-a3595a4cd847-model-cache\") pod \"distinct-w3-5a8f94-kserve-5f8b544d59-mf95j\" (UID: \"b8ac1cca-4ebd-474c-b32f-a3595a4cd847\") " pod="e2e-models-e2e-worker-w3-5a8f94/distinct-w3-5a8f94-kserve-5f8b544d59-mf95j"
Oct 30 21:00:03.938773 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:03.938657    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8ac1cca-4ebd-474c-b32f-a3595a4cd847-tmp-dir\") pod \"distinct-w3-5a8f94-kserve-5f8b544d59-mf95j\" (UID: \"b8ac1cca-4ebd-474c-b32f-a3595a4cd847\") " pod="e2e-models-e2e-worker-w3-5a8f94/distinct-w3-5a8f94-kserve-5f8b544d59-mf95j"
Oct 30 21:00:03.939931 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:03.939562    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8ac1cca-4ebd-474c-b32f-a3595a4cd847-tmp-dir\") pod \"distinct-w3-5a8f94-kserve-5f8b544d59-mf95j\" (UID: \"b8ac1cca-4ebd-474c-b32f-a3595a4cd847\") " pod="e2e-models-e2e-worker-w3-5a8f94/distinct-w3-5a8f94-kserve-5f8b544d59-mf95j"
Oct 30 21:00:03.939931 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:03.939855    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8ac1cca-4ebd-474c-b32f-a3595a4cd847-home\") pod \"distinct-w3-5a8f94-kserve-5f8b544d59-mf95j\" (UID: \"b8ac1cca-4ebd-474c-b32f-a3595a4cd847\") " pod="e2e-models-e2e-worker-w3-5a8f94/distinct-w3-5a8f94-kserve-5f8b544d59-mf95j"
Oct 30 21:00:03.939931 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:03.939874    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8ac1cca-4ebd-474c-b32f-a3595a4cd847-model-cache\") pod \"distinct-w3-5a8f94-kserve-5f8b544d59-mf95j\" (UID: \"b8ac1cca-4ebd-474c-b32f-a3595a4cd847\") " pod="e2e-models-e2e-worker-w3-5a8f94/distinct-w3-5a8f94-kserve-5f8b544d59-mf95j"
Oct 30 21:00:03.942825 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:03.942732    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8ac1cca-4ebd-474c-b32f-a3595a4cd847-tls-certs\") pod \"distinct-w3-5a8f94-kserve-5f8b544d59-mf95j\" (UID: \"b8ac1cca-4ebd-474c-b32f-a3595a4cd847\") " pod="e2e-models-e2e-worker-w3-5a8f94/distinct-w3-5a8f94-kserve-5f8b544d59-mf95j"
Oct 30 21:00:03.943763 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:03.943735    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8ac1cca-4ebd-474c-b32f-a3595a4cd847-dshm\") pod \"distinct-w3-5a8f94-kserve-5f8b544d59-mf95j\" (UID: \"b8ac1cca-4ebd-474c-b32f-a3595a4cd847\") " pod="e2e-models-e2e-worker-w3-5a8f94/distinct-w3-5a8f94-kserve-5f8b544d59-mf95j"
Oct 30 21:00:03.957790 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:03.957756    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bstjz\" (UniqueName: \"kubernetes.io/projected/b8ac1cca-4ebd-474c-b32f-a3595a4cd847-kube-api-access-bstjz\") pod \"distinct-w3-5a8f94-kserve-5f8b544d59-mf95j\" (UID: \"b8ac1cca-4ebd-474c-b32f-a3595a4cd847\") " pod="e2e-models-e2e-worker-w3-5a8f94/distinct-w3-5a8f94-kserve-5f8b544d59-mf95j"
Oct 30 21:00:04.113382 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:04.113346    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-5a8f94/distinct-w3-5a8f94-kserve-5f8b544d59-mf95j"
Oct 30 21:00:04.397562 ip-10-0-131-125 kubenswrapper[4521]: W1030 21:00:04.397532    4521 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb8ac1cca_4ebd_474c_b32f_a3595a4cd847.slice/crio-b7e094cf578f7e7de41fa75caeab6a454f555740d869f20455f6b1c795712b83 WatchSource:0}: Error finding container b7e094cf578f7e7de41fa75caeab6a454f555740d869f20455f6b1c795712b83: Status 404 returned error can't find the container with id b7e094cf578f7e7de41fa75caeab6a454f555740d869f20455f6b1c795712b83
Oct 30 21:00:04.398922 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:04.398894    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-5a8f94/distinct-w3-5a8f94-kserve-5f8b544d59-mf95j"]
Oct 30 21:00:04.711428 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:04.711393    4521 generic.go:356] "Generic (PLEG): container finished" podID="f0bc9c01-1d10-4e3d-9876-f24637b968ef" containerID="b42a19e8e609608a0cf9a112877cc76091fe4dcada28ce1fb6cc83b85f77222c" exitCode=60
Oct 30 21:00:04.711925 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:04.711508    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-65968b-29889900-hkfkj" event={"ID":"f0bc9c01-1d10-4e3d-9876-f24637b968ef","Type":"ContainerDied","Data":"b42a19e8e609608a0cf9a112877cc76091fe4dcada28ce1fb6cc83b85f77222c"}
Oct 30 21:00:04.711925 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:04.711579    4521 scope.go:122] "RemoveContainer" containerID="ef8964114fe1841fddea74799dfc5b265df85802ad9ea372d915ca67ecbb1e35"
Oct 30 21:00:04.711925 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:04.711790    4521 scope.go:122] "RemoveContainer" containerID="b42a19e8e609608a0cf9a112877cc76091fe4dcada28ce1fb6cc83b85f77222c"
Oct 30 21:00:04.712130 ip-10-0-131-125 kubenswrapper[4521]: E1030 21:00:04.712107    4521 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-65968b-29889900-hkfkj_odh-ai-gateway-infra(f0bc9c01-1d10-4e3d-9876-f24637b968ef)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-65968b-29889900-hkfkj" podUID="f0bc9c01-1d10-4e3d-9876-f24637b968ef"
Oct 30 21:00:04.714711 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:04.714687    4521 generic.go:356] "Generic (PLEG): container finished" podID="1e319e8b-6373-4ad3-b549-086f4a0c0f25" containerID="8354941073a53fcc812e0428dd49e48225c40acf3f7f36e0aefca53a4060fbbe" exitCode=60
Oct 30 21:00:04.714808 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:04.714760    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c4b379-29889900-sbvxl" event={"ID":"1e319e8b-6373-4ad3-b549-086f4a0c0f25","Type":"ContainerDied","Data":"8354941073a53fcc812e0428dd49e48225c40acf3f7f36e0aefca53a4060fbbe"}
Oct 30 21:00:04.715073 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:04.715054    4521 scope.go:122] "RemoveContainer" containerID="8354941073a53fcc812e0428dd49e48225c40acf3f7f36e0aefca53a4060fbbe"
Oct 30 21:00:04.715353 ip-10-0-131-125 kubenswrapper[4521]: E1030 21:00:04.715315    4521 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-c4b379-29889900-sbvxl_odh-ai-gateway-infra(1e319e8b-6373-4ad3-b549-086f4a0c0f25)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c4b379-29889900-sbvxl" podUID="1e319e8b-6373-4ad3-b549-086f4a0c0f25"
Oct 30 21:00:04.717928 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:04.717303    4521 scope.go:122] "RemoveContainer" containerID="7be1b30dd65665678dbc107591759f5cd9a166ad0356e7fae892af4552330fb3"
Oct 30 21:00:04.717928 ip-10-0-131-125 kubenswrapper[4521]: E1030 21:00:04.717624    4521 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-39fb96-29889900-sbp6w_odh-ai-gateway-infra(49b3ef08-eca0-4a62-885d-bcf8fc23cb6d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-39fb96-29889900-sbp6w" podUID="49b3ef08-eca0-4a62-885d-bcf8fc23cb6d"
Oct 30 21:00:04.719451 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:04.719376    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-5a8f94/distinct-w3-5a8f94-kserve-5f8b544d59-mf95j" event={"ID":"b8ac1cca-4ebd-474c-b32f-a3595a4cd847","Type":"ContainerStarted","Data":"ec30aab6c0ae6126f6e198b644f2bcc5738d6823efd7ef36cca8326416a01eb2"}
Oct 30 21:00:04.719451 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:04.719414    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-5a8f94/distinct-w3-5a8f94-kserve-5f8b544d59-mf95j" event={"ID":"b8ac1cca-4ebd-474c-b32f-a3595a4cd847","Type":"ContainerStarted","Data":"b7e094cf578f7e7de41fa75caeab6a454f555740d869f20455f6b1c795712b83"}
Oct 30 21:00:04.727926 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:04.727903    4521 scope.go:122] "RemoveContainer" containerID="6357f80f48a77164ea54261b2a461394c6e0338290f840ee7aaa8d67c2539412"
Oct 30 21:00:04.781885 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:04.781831    4521 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-5a8f94/distinct-w3-5a8f94-kserve-5f8b544d59-mf95j" podStartSLOduration=1.781816007 podStartE2EDuration="1.781816007s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:00:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:00:04.778426889 +0000 UTC m=+1272.124048122" watchObservedRunningTime="2026-10-30 21:00:04.781816007 +0000 UTC m=+1272.127437275"
Oct 30 21:00:05.113636 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:05.113542    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5a8f94/distinct-w3-5a8f94-kserve-5f8b544d59-mf95j"
Oct 30 21:00:05.729608 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:05.729576    4521 scope.go:122] "RemoveContainer" containerID="b42a19e8e609608a0cf9a112877cc76091fe4dcada28ce1fb6cc83b85f77222c"
Oct 30 21:00:05.730065 ip-10-0-131-125 kubenswrapper[4521]: E1030 21:00:05.729893    4521 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-65968b-29889900-hkfkj_odh-ai-gateway-infra(f0bc9c01-1d10-4e3d-9876-f24637b968ef)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-65968b-29889900-hkfkj" podUID="f0bc9c01-1d10-4e3d-9876-f24637b968ef"
Oct 30 21:00:05.731491 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:05.731467    4521 scope.go:122] "RemoveContainer" containerID="8354941073a53fcc812e0428dd49e48225c40acf3f7f36e0aefca53a4060fbbe"
Oct 30 21:00:05.731762 ip-10-0-131-125 kubenswrapper[4521]: E1030 21:00:05.731739    4521 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-c4b379-29889900-sbvxl_odh-ai-gateway-infra(1e319e8b-6373-4ad3-b549-086f4a0c0f25)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c4b379-29889900-sbvxl" podUID="1e319e8b-6373-4ad3-b549-086f4a0c0f25"
Oct 30 21:00:06.722261 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:06.722223    4521 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-32a969/distinct-w2-32a969-kserve-7f798b664f-jdh82"]
Oct 30 21:00:06.727819 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:06.727790    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-32a969/distinct-w2-32a969-kserve-7f798b664f-jdh82"
Oct 30 21:00:06.736415 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:06.736383    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-32a969/distinct-w2-32a969-kserve-7f798b664f-jdh82"]
Oct 30 21:00:06.751249 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:06.751206    4521 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-32a969\"/\"distinct-w2-32a969-kserve-self-signed-certs\""
Oct 30 21:00:06.769824 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:06.769787    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/887f1d27-bf24-4b57-b6e4-8fb095df7a50-dshm\") pod \"distinct-w2-32a969-kserve-7f798b664f-jdh82\" (UID: \"887f1d27-bf24-4b57-b6e4-8fb095df7a50\") " pod="e2e-models-e2e-worker-w2-32a969/distinct-w2-32a969-kserve-7f798b664f-jdh82"
Oct 30 21:00:06.770023 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:06.769842    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvs47\" (UniqueName: \"kubernetes.io/projected/887f1d27-bf24-4b57-b6e4-8fb095df7a50-kube-api-access-mvs47\") pod \"distinct-w2-32a969-kserve-7f798b664f-jdh82\" (UID: \"887f1d27-bf24-4b57-b6e4-8fb095df7a50\") " pod="e2e-models-e2e-worker-w2-32a969/distinct-w2-32a969-kserve-7f798b664f-jdh82"
Oct 30 21:00:06.770023 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:06.769882    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/887f1d27-bf24-4b57-b6e4-8fb095df7a50-model-cache\") pod \"distinct-w2-32a969-kserve-7f798b664f-jdh82\" (UID: \"887f1d27-bf24-4b57-b6e4-8fb095df7a50\") " pod="e2e-models-e2e-worker-w2-32a969/distinct-w2-32a969-kserve-7f798b664f-jdh82"
Oct 30 21:00:06.770023 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:06.769907    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/887f1d27-bf24-4b57-b6e4-8fb095df7a50-home\") pod \"distinct-w2-32a969-kserve-7f798b664f-jdh82\" (UID: \"887f1d27-bf24-4b57-b6e4-8fb095df7a50\") " pod="e2e-models-e2e-worker-w2-32a969/distinct-w2-32a969-kserve-7f798b664f-jdh82"
Oct 30 21:00:06.770023 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:06.769966    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/887f1d27-bf24-4b57-b6e4-8fb095df7a50-tls-certs\") pod \"distinct-w2-32a969-kserve-7f798b664f-jdh82\" (UID: \"887f1d27-bf24-4b57-b6e4-8fb095df7a50\") " pod="e2e-models-e2e-worker-w2-32a969/distinct-w2-32a969-kserve-7f798b664f-jdh82"
Oct 30 21:00:06.770259 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:06.770099    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/887f1d27-bf24-4b57-b6e4-8fb095df7a50-tmp-dir\") pod \"distinct-w2-32a969-kserve-7f798b664f-jdh82\" (UID: \"887f1d27-bf24-4b57-b6e4-8fb095df7a50\") " pod="e2e-models-e2e-worker-w2-32a969/distinct-w2-32a969-kserve-7f798b664f-jdh82"
Oct 30 21:00:06.890342 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:06.880037    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/887f1d27-bf24-4b57-b6e4-8fb095df7a50-tmp-dir\") pod \"distinct-w2-32a969-kserve-7f798b664f-jdh82\" (UID: \"887f1d27-bf24-4b57-b6e4-8fb095df7a50\") " pod="e2e-models-e2e-worker-w2-32a969/distinct-w2-32a969-kserve-7f798b664f-jdh82"
Oct 30 21:00:06.890342 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:06.880120    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/887f1d27-bf24-4b57-b6e4-8fb095df7a50-dshm\") pod \"distinct-w2-32a969-kserve-7f798b664f-jdh82\" (UID: \"887f1d27-bf24-4b57-b6e4-8fb095df7a50\") " pod="e2e-models-e2e-worker-w2-32a969/distinct-w2-32a969-kserve-7f798b664f-jdh82"
Oct 30 21:00:06.890342 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:06.880157    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvs47\" (UniqueName: \"kubernetes.io/projected/887f1d27-bf24-4b57-b6e4-8fb095df7a50-kube-api-access-mvs47\") pod \"distinct-w2-32a969-kserve-7f798b664f-jdh82\" (UID: \"887f1d27-bf24-4b57-b6e4-8fb095df7a50\") " pod="e2e-models-e2e-worker-w2-32a969/distinct-w2-32a969-kserve-7f798b664f-jdh82"
Oct 30 21:00:06.890342 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:06.880194    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/887f1d27-bf24-4b57-b6e4-8fb095df7a50-model-cache\") pod \"distinct-w2-32a969-kserve-7f798b664f-jdh82\" (UID: \"887f1d27-bf24-4b57-b6e4-8fb095df7a50\") " pod="e2e-models-e2e-worker-w2-32a969/distinct-w2-32a969-kserve-7f798b664f-jdh82"
Oct 30 21:00:06.890342 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:06.880219    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/887f1d27-bf24-4b57-b6e4-8fb095df7a50-home\") pod \"distinct-w2-32a969-kserve-7f798b664f-jdh82\" (UID: \"887f1d27-bf24-4b57-b6e4-8fb095df7a50\") " pod="e2e-models-e2e-worker-w2-32a969/distinct-w2-32a969-kserve-7f798b664f-jdh82"
Oct 30 21:00:06.890342 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:06.880288    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/887f1d27-bf24-4b57-b6e4-8fb095df7a50-tls-certs\") pod \"distinct-w2-32a969-kserve-7f798b664f-jdh82\" (UID: \"887f1d27-bf24-4b57-b6e4-8fb095df7a50\") " pod="e2e-models-e2e-worker-w2-32a969/distinct-w2-32a969-kserve-7f798b664f-jdh82"
Oct 30 21:00:06.890342 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:06.881817    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/887f1d27-bf24-4b57-b6e4-8fb095df7a50-tmp-dir\") pod \"distinct-w2-32a969-kserve-7f798b664f-jdh82\" (UID: \"887f1d27-bf24-4b57-b6e4-8fb095df7a50\") " pod="e2e-models-e2e-worker-w2-32a969/distinct-w2-32a969-kserve-7f798b664f-jdh82"
Oct 30 21:00:06.890342 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:06.882091    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/887f1d27-bf24-4b57-b6e4-8fb095df7a50-model-cache\") pod \"distinct-w2-32a969-kserve-7f798b664f-jdh82\" (UID: \"887f1d27-bf24-4b57-b6e4-8fb095df7a50\") " pod="e2e-models-e2e-worker-w2-32a969/distinct-w2-32a969-kserve-7f798b664f-jdh82"
Oct 30 21:00:06.890342 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:06.882317    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/887f1d27-bf24-4b57-b6e4-8fb095df7a50-home\") pod \"distinct-w2-32a969-kserve-7f798b664f-jdh82\" (UID: \"887f1d27-bf24-4b57-b6e4-8fb095df7a50\") " pod="e2e-models-e2e-worker-w2-32a969/distinct-w2-32a969-kserve-7f798b664f-jdh82"
Oct 30 21:00:06.890342 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:06.885022    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/887f1d27-bf24-4b57-b6e4-8fb095df7a50-dshm\") pod \"distinct-w2-32a969-kserve-7f798b664f-jdh82\" (UID: \"887f1d27-bf24-4b57-b6e4-8fb095df7a50\") " pod="e2e-models-e2e-worker-w2-32a969/distinct-w2-32a969-kserve-7f798b664f-jdh82"
Oct 30 21:00:06.897377 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:06.894269    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/887f1d27-bf24-4b57-b6e4-8fb095df7a50-tls-certs\") pod \"distinct-w2-32a969-kserve-7f798b664f-jdh82\" (UID: \"887f1d27-bf24-4b57-b6e4-8fb095df7a50\") " pod="e2e-models-e2e-worker-w2-32a969/distinct-w2-32a969-kserve-7f798b664f-jdh82"
Oct 30 21:00:06.909151 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:06.909022    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvs47\" (UniqueName: \"kubernetes.io/projected/887f1d27-bf24-4b57-b6e4-8fb095df7a50-kube-api-access-mvs47\") pod \"distinct-w2-32a969-kserve-7f798b664f-jdh82\" (UID: \"887f1d27-bf24-4b57-b6e4-8fb095df7a50\") " pod="e2e-models-e2e-worker-w2-32a969/distinct-w2-32a969-kserve-7f798b664f-jdh82"
Oct 30 21:00:07.053563 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:07.053473    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-32a969/distinct-w2-32a969-kserve-7f798b664f-jdh82"
Oct 30 21:00:07.340536 ip-10-0-131-125 kubenswrapper[4521]: W1030 21:00:07.340506    4521 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod887f1d27_bf24_4b57_b6e4_8fb095df7a50.slice/crio-d61100423af8a92e7821e7250379d5e8a3b7a7d9802b449ad1729b9a5f6aac6d WatchSource:0}: Error finding container d61100423af8a92e7821e7250379d5e8a3b7a7d9802b449ad1729b9a5f6aac6d: Status 404 returned error can't find the container with id d61100423af8a92e7821e7250379d5e8a3b7a7d9802b449ad1729b9a5f6aac6d
Oct 30 21:00:07.341422 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:07.341377    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-32a969/distinct-w2-32a969-kserve-7f798b664f-jdh82"]
Oct 30 21:00:07.778136 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:07.778094    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-32a969/distinct-w2-32a969-kserve-7f798b664f-jdh82" event={"ID":"887f1d27-bf24-4b57-b6e4-8fb095df7a50","Type":"ContainerStarted","Data":"46781044a82ac81762140ed527675663b0e3cda210817c2aa5dbb28004f922b6"}
Oct 30 21:00:07.778783 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:07.778760    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-32a969/distinct-w2-32a969-kserve-7f798b664f-jdh82" event={"ID":"887f1d27-bf24-4b57-b6e4-8fb095df7a50","Type":"ContainerStarted","Data":"d61100423af8a92e7821e7250379d5e8a3b7a7d9802b449ad1729b9a5f6aac6d"}
Oct 30 21:00:07.799682 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:07.799609    4521 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-32a969/distinct-w2-32a969-kserve-7f798b664f-jdh82" podStartSLOduration=1.7995917160000001 podStartE2EDuration="1.799591716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:00:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:00:07.797909036 +0000 UTC m=+1275.143530277" watchObservedRunningTime="2026-10-30 21:00:07.799591716 +0000 UTC m=+1275.145212955"
Oct 30 21:00:08.056529 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:08.054177    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-32a969/distinct-w2-32a969-kserve-7f798b664f-jdh82"
Oct 30 21:00:10.547062 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:10.547013    4521 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-tqd8s"]
Oct 30 21:00:10.551041 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:10.551009    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-tqd8s"
Oct 30 21:00:10.557539 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:10.557506    4521 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-32a969-openshift-default-dockercfg-tkmt5\""
Oct 30 21:00:10.565931 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:10.565904    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-tqd8s"]
Oct 30 21:00:10.652629 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:10.652591    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00b6adb9-3733-4ea4-9a26-f3b29991001a-istio-podinfo\") pod \"e2e-worker-w2-32a969-openshift-default-8945d87fc-tqd8s\" (UID: \"00b6adb9-3733-4ea4-9a26-f3b29991001a\") " pod="openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-tqd8s"
Oct 30 21:00:10.652629 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:10.652632    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00b6adb9-3733-4ea4-9a26-f3b29991001a-istiod-ca-cert\") pod \"e2e-worker-w2-32a969-openshift-default-8945d87fc-tqd8s\" (UID: \"00b6adb9-3733-4ea4-9a26-f3b29991001a\") " pod="openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-tqd8s"
Oct 30 21:00:10.652883 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:10.652653    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00b6adb9-3733-4ea4-9a26-f3b29991001a-istio-data\") pod \"e2e-worker-w2-32a969-openshift-default-8945d87fc-tqd8s\" (UID: \"00b6adb9-3733-4ea4-9a26-f3b29991001a\") " pod="openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-tqd8s"
Oct 30 21:00:10.652883 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:10.652777    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00b6adb9-3733-4ea4-9a26-f3b29991001a-credential-socket\") pod \"e2e-worker-w2-32a969-openshift-default-8945d87fc-tqd8s\" (UID: \"00b6adb9-3733-4ea4-9a26-f3b29991001a\") " pod="openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-tqd8s"
Oct 30 21:00:10.652883 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:10.652816    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00b6adb9-3733-4ea4-9a26-f3b29991001a-istio-token\") pod \"e2e-worker-w2-32a969-openshift-default-8945d87fc-tqd8s\" (UID: \"00b6adb9-3733-4ea4-9a26-f3b29991001a\") " pod="openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-tqd8s"
Oct 30 21:00:10.653051 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:10.652915    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmrvp\" (UniqueName: \"kubernetes.io/projected/00b6adb9-3733-4ea4-9a26-f3b29991001a-kube-api-access-xmrvp\") pod \"e2e-worker-w2-32a969-openshift-default-8945d87fc-tqd8s\" (UID: \"00b6adb9-3733-4ea4-9a26-f3b29991001a\") " pod="openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-tqd8s"
Oct 30 21:00:10.653051 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:10.653008    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00b6adb9-3733-4ea4-9a26-f3b29991001a-workload-certs\") pod \"e2e-worker-w2-32a969-openshift-default-8945d87fc-tqd8s\" (UID: \"00b6adb9-3733-4ea4-9a26-f3b29991001a\") " pod="openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-tqd8s"
Oct 30 21:00:10.653051 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:10.653039    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00b6adb9-3733-4ea4-9a26-f3b29991001a-workload-socket\") pod \"e2e-worker-w2-32a969-openshift-default-8945d87fc-tqd8s\" (UID: \"00b6adb9-3733-4ea4-9a26-f3b29991001a\") " pod="openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-tqd8s"
Oct 30 21:00:10.653210 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:10.653107    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00b6adb9-3733-4ea4-9a26-f3b29991001a-istio-envoy\") pod \"e2e-worker-w2-32a969-openshift-default-8945d87fc-tqd8s\" (UID: \"00b6adb9-3733-4ea4-9a26-f3b29991001a\") " pod="openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-tqd8s"
Oct 30 21:00:10.754461 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:10.754426    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00b6adb9-3733-4ea4-9a26-f3b29991001a-workload-certs\") pod \"e2e-worker-w2-32a969-openshift-default-8945d87fc-tqd8s\" (UID: \"00b6adb9-3733-4ea4-9a26-f3b29991001a\") " pod="openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-tqd8s"
Oct 30 21:00:10.754461 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:10.754465    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00b6adb9-3733-4ea4-9a26-f3b29991001a-workload-socket\") pod \"e2e-worker-w2-32a969-openshift-default-8945d87fc-tqd8s\" (UID: \"00b6adb9-3733-4ea4-9a26-f3b29991001a\") " pod="openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-tqd8s"
Oct 30 21:00:10.754730 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:10.754510    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00b6adb9-3733-4ea4-9a26-f3b29991001a-istio-envoy\") pod \"e2e-worker-w2-32a969-openshift-default-8945d87fc-tqd8s\" (UID: \"00b6adb9-3733-4ea4-9a26-f3b29991001a\") " pod="openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-tqd8s"
Oct 30 21:00:10.754730 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:10.754551    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00b6adb9-3733-4ea4-9a26-f3b29991001a-istio-podinfo\") pod \"e2e-worker-w2-32a969-openshift-default-8945d87fc-tqd8s\" (UID: \"00b6adb9-3733-4ea4-9a26-f3b29991001a\") " pod="openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-tqd8s"
Oct 30 21:00:10.754730 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:10.754693    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00b6adb9-3733-4ea4-9a26-f3b29991001a-istiod-ca-cert\") pod \"e2e-worker-w2-32a969-openshift-default-8945d87fc-tqd8s\" (UID: \"00b6adb9-3733-4ea4-9a26-f3b29991001a\") " pod="openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-tqd8s"
Oct 30 21:00:10.754890 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:10.754743    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00b6adb9-3733-4ea4-9a26-f3b29991001a-istio-data\") pod \"e2e-worker-w2-32a969-openshift-default-8945d87fc-tqd8s\" (UID: \"00b6adb9-3733-4ea4-9a26-f3b29991001a\") " pod="openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-tqd8s"
Oct 30 21:00:10.754890 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:10.754804    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00b6adb9-3733-4ea4-9a26-f3b29991001a-credential-socket\") pod \"e2e-worker-w2-32a969-openshift-default-8945d87fc-tqd8s\" (UID: \"00b6adb9-3733-4ea4-9a26-f3b29991001a\") " pod="openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-tqd8s"
Oct 30 21:00:10.754890 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:10.754862    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00b6adb9-3733-4ea4-9a26-f3b29991001a-istio-token\") pod \"e2e-worker-w2-32a969-openshift-default-8945d87fc-tqd8s\" (UID: \"00b6adb9-3733-4ea4-9a26-f3b29991001a\") " pod="openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-tqd8s"
Oct 30 21:00:10.755058 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:10.754900    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00b6adb9-3733-4ea4-9a26-f3b29991001a-workload-certs\") pod \"e2e-worker-w2-32a969-openshift-default-8945d87fc-tqd8s\" (UID: \"00b6adb9-3733-4ea4-9a26-f3b29991001a\") " pod="openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-tqd8s"
Oct 30 21:00:10.755058 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:10.754908    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmrvp\" (UniqueName: \"kubernetes.io/projected/00b6adb9-3733-4ea4-9a26-f3b29991001a-kube-api-access-xmrvp\") pod \"e2e-worker-w2-32a969-openshift-default-8945d87fc-tqd8s\" (UID: \"00b6adb9-3733-4ea4-9a26-f3b29991001a\") " pod="openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-tqd8s"
Oct 30 21:00:10.755058 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:10.754969    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00b6adb9-3733-4ea4-9a26-f3b29991001a-workload-socket\") pod \"e2e-worker-w2-32a969-openshift-default-8945d87fc-tqd8s\" (UID: \"00b6adb9-3733-4ea4-9a26-f3b29991001a\") " pod="openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-tqd8s"
Oct 30 21:00:10.755404 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:10.755380    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00b6adb9-3733-4ea4-9a26-f3b29991001a-credential-socket\") pod \"e2e-worker-w2-32a969-openshift-default-8945d87fc-tqd8s\" (UID: \"00b6adb9-3733-4ea4-9a26-f3b29991001a\") " pod="openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-tqd8s"
Oct 30 21:00:10.755565 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:10.755540    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00b6adb9-3733-4ea4-9a26-f3b29991001a-istio-data\") pod \"e2e-worker-w2-32a969-openshift-default-8945d87fc-tqd8s\" (UID: \"00b6adb9-3733-4ea4-9a26-f3b29991001a\") " pod="openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-tqd8s"
Oct 30 21:00:10.755733 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:10.755706    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00b6adb9-3733-4ea4-9a26-f3b29991001a-istiod-ca-cert\") pod \"e2e-worker-w2-32a969-openshift-default-8945d87fc-tqd8s\" (UID: \"00b6adb9-3733-4ea4-9a26-f3b29991001a\") " pod="openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-tqd8s"
Oct 30 21:00:10.757162 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:10.757138    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00b6adb9-3733-4ea4-9a26-f3b29991001a-istio-envoy\") pod \"e2e-worker-w2-32a969-openshift-default-8945d87fc-tqd8s\" (UID: \"00b6adb9-3733-4ea4-9a26-f3b29991001a\") " pod="openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-tqd8s"
Oct 30 21:00:10.757363 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:10.757343    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00b6adb9-3733-4ea4-9a26-f3b29991001a-istio-podinfo\") pod \"e2e-worker-w2-32a969-openshift-default-8945d87fc-tqd8s\" (UID: \"00b6adb9-3733-4ea4-9a26-f3b29991001a\") " pod="openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-tqd8s"
Oct 30 21:00:10.779370 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:10.771798    4521 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-6ggs9"]
Oct 30 21:00:10.779563 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:10.779373    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-6ggs9"
Oct 30 21:00:10.780863 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:10.780837    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00b6adb9-3733-4ea4-9a26-f3b29991001a-istio-token\") pod \"e2e-worker-w2-32a969-openshift-default-8945d87fc-tqd8s\" (UID: \"00b6adb9-3733-4ea4-9a26-f3b29991001a\") " pod="openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-tqd8s"
Oct 30 21:00:10.781880 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:10.781857    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmrvp\" (UniqueName: \"kubernetes.io/projected/00b6adb9-3733-4ea4-9a26-f3b29991001a-kube-api-access-xmrvp\") pod \"e2e-worker-w2-32a969-openshift-default-8945d87fc-tqd8s\" (UID: \"00b6adb9-3733-4ea4-9a26-f3b29991001a\") " pod="openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-tqd8s"
Oct 30 21:00:10.784897 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:10.784870    4521 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-c2d1eb-openshift-default-dockercfg-x4gzl\""
Oct 30 21:00:10.814378 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:10.811838    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-6ggs9"]
Oct 30 21:00:10.857760 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:10.857725    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/795d7e19-cf6e-4863-bf80-28ae03af6a98-workload-socket\") pod \"e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-6ggs9\" (UID: \"795d7e19-cf6e-4863-bf80-28ae03af6a98\") " pod="openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-6ggs9"
Oct 30 21:00:10.857944 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:10.857775    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/795d7e19-cf6e-4863-bf80-28ae03af6a98-istio-envoy\") pod \"e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-6ggs9\" (UID: \"795d7e19-cf6e-4863-bf80-28ae03af6a98\") " pod="openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-6ggs9"
Oct 30 21:00:10.857944 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:10.857802    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/795d7e19-cf6e-4863-bf80-28ae03af6a98-istio-data\") pod \"e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-6ggs9\" (UID: \"795d7e19-cf6e-4863-bf80-28ae03af6a98\") " pod="openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-6ggs9"
Oct 30 21:00:10.857944 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:10.857899    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/795d7e19-cf6e-4863-bf80-28ae03af6a98-istiod-ca-cert\") pod \"e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-6ggs9\" (UID: \"795d7e19-cf6e-4863-bf80-28ae03af6a98\") " pod="openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-6ggs9"
Oct 30 21:00:10.858137 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:10.857971    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/795d7e19-cf6e-4863-bf80-28ae03af6a98-istio-podinfo\") pod \"e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-6ggs9\" (UID: \"795d7e19-cf6e-4863-bf80-28ae03af6a98\") " pod="openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-6ggs9"
Oct 30 21:00:10.858137 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:10.858044    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/795d7e19-cf6e-4863-bf80-28ae03af6a98-workload-certs\") pod \"e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-6ggs9\" (UID: \"795d7e19-cf6e-4863-bf80-28ae03af6a98\") " pod="openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-6ggs9"
Oct 30 21:00:10.858242 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:10.858179    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/795d7e19-cf6e-4863-bf80-28ae03af6a98-credential-socket\") pod \"e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-6ggs9\" (UID: \"795d7e19-cf6e-4863-bf80-28ae03af6a98\") " pod="openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-6ggs9"
Oct 30 21:00:10.858242 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:10.858226    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/795d7e19-cf6e-4863-bf80-28ae03af6a98-istio-token\") pod \"e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-6ggs9\" (UID: \"795d7e19-cf6e-4863-bf80-28ae03af6a98\") " pod="openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-6ggs9"
Oct 30 21:00:10.858406 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:10.858387    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rplp\" (UniqueName: \"kubernetes.io/projected/795d7e19-cf6e-4863-bf80-28ae03af6a98-kube-api-access-2rplp\") pod \"e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-6ggs9\" (UID: \"795d7e19-cf6e-4863-bf80-28ae03af6a98\") " pod="openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-6ggs9"
Oct 30 21:00:10.868502 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:10.868468    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-tqd8s"
Oct 30 21:00:10.961920 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:10.959872    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rplp\" (UniqueName: \"kubernetes.io/projected/795d7e19-cf6e-4863-bf80-28ae03af6a98-kube-api-access-2rplp\") pod \"e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-6ggs9\" (UID: \"795d7e19-cf6e-4863-bf80-28ae03af6a98\") " pod="openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-6ggs9"
Oct 30 21:00:10.961920 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:10.959925    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/795d7e19-cf6e-4863-bf80-28ae03af6a98-workload-socket\") pod \"e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-6ggs9\" (UID: \"795d7e19-cf6e-4863-bf80-28ae03af6a98\") " pod="openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-6ggs9"
Oct 30 21:00:10.961920 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:10.959944    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/795d7e19-cf6e-4863-bf80-28ae03af6a98-istio-envoy\") pod \"e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-6ggs9\" (UID: \"795d7e19-cf6e-4863-bf80-28ae03af6a98\") " pod="openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-6ggs9"
Oct 30 21:00:10.961920 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:10.959958    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/795d7e19-cf6e-4863-bf80-28ae03af6a98-istio-data\") pod \"e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-6ggs9\" (UID: \"795d7e19-cf6e-4863-bf80-28ae03af6a98\") " pod="openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-6ggs9"
Oct 30 21:00:10.961920 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:10.959979    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/795d7e19-cf6e-4863-bf80-28ae03af6a98-istiod-ca-cert\") pod \"e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-6ggs9\" (UID: \"795d7e19-cf6e-4863-bf80-28ae03af6a98\") " pod="openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-6ggs9"
Oct 30 21:00:10.961920 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:10.960003    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/795d7e19-cf6e-4863-bf80-28ae03af6a98-istio-podinfo\") pod \"e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-6ggs9\" (UID: \"795d7e19-cf6e-4863-bf80-28ae03af6a98\") " pod="openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-6ggs9"
Oct 30 21:00:10.961920 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:10.960039    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/795d7e19-cf6e-4863-bf80-28ae03af6a98-workload-certs\") pod \"e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-6ggs9\" (UID: \"795d7e19-cf6e-4863-bf80-28ae03af6a98\") " pod="openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-6ggs9"
Oct 30 21:00:10.961920 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:10.960088    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/795d7e19-cf6e-4863-bf80-28ae03af6a98-credential-socket\") pod \"e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-6ggs9\" (UID: \"795d7e19-cf6e-4863-bf80-28ae03af6a98\") " pod="openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-6ggs9"
Oct 30 21:00:10.961920 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:10.960114    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/795d7e19-cf6e-4863-bf80-28ae03af6a98-istio-token\") pod \"e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-6ggs9\" (UID: \"795d7e19-cf6e-4863-bf80-28ae03af6a98\") " pod="openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-6ggs9"
Oct 30 21:00:10.961920 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:10.960825    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/795d7e19-cf6e-4863-bf80-28ae03af6a98-workload-socket\") pod \"e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-6ggs9\" (UID: \"795d7e19-cf6e-4863-bf80-28ae03af6a98\") " pod="openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-6ggs9"
Oct 30 21:00:10.961920 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:10.961453    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/795d7e19-cf6e-4863-bf80-28ae03af6a98-istio-data\") pod \"e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-6ggs9\" (UID: \"795d7e19-cf6e-4863-bf80-28ae03af6a98\") " pod="openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-6ggs9"
Oct 30 21:00:10.961920 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:10.961714    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/795d7e19-cf6e-4863-bf80-28ae03af6a98-workload-certs\") pod \"e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-6ggs9\" (UID: \"795d7e19-cf6e-4863-bf80-28ae03af6a98\") " pod="openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-6ggs9"
Oct 30 21:00:10.961920 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:10.961752    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/795d7e19-cf6e-4863-bf80-28ae03af6a98-credential-socket\") pod \"e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-6ggs9\" (UID: \"795d7e19-cf6e-4863-bf80-28ae03af6a98\") " pod="openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-6ggs9"
Oct 30 21:00:10.962655 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:10.962358    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/795d7e19-cf6e-4863-bf80-28ae03af6a98-istiod-ca-cert\") pod \"e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-6ggs9\" (UID: \"795d7e19-cf6e-4863-bf80-28ae03af6a98\") " pod="openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-6ggs9"
Oct 30 21:00:10.964695 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:10.964035    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/795d7e19-cf6e-4863-bf80-28ae03af6a98-istio-podinfo\") pod \"e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-6ggs9\" (UID: \"795d7e19-cf6e-4863-bf80-28ae03af6a98\") " pod="openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-6ggs9"
Oct 30 21:00:10.964695 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:10.964505    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/795d7e19-cf6e-4863-bf80-28ae03af6a98-istio-envoy\") pod \"e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-6ggs9\" (UID: \"795d7e19-cf6e-4863-bf80-28ae03af6a98\") " pod="openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-6ggs9"
Oct 30 21:00:10.988723 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:10.988659    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/795d7e19-cf6e-4863-bf80-28ae03af6a98-istio-token\") pod \"e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-6ggs9\" (UID: \"795d7e19-cf6e-4863-bf80-28ae03af6a98\") " pod="openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-6ggs9"
Oct 30 21:00:10.997401 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:10.997370    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rplp\" (UniqueName: \"kubernetes.io/projected/795d7e19-cf6e-4863-bf80-28ae03af6a98-kube-api-access-2rplp\") pod \"e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-6ggs9\" (UID: \"795d7e19-cf6e-4863-bf80-28ae03af6a98\") " pod="openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-6ggs9"
Oct 30 21:00:11.005839 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:11.005812    4521 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-lp6sk"]
Oct 30 21:00:11.012575 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:11.012550    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-lp6sk"
Oct 30 21:00:11.037345 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:11.033921    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-lp6sk"]
Oct 30 21:00:11.045367 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:11.045168    4521 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-5xzm6"]
Oct 30 21:00:11.049806 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:11.049746    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-5xzm6"
Oct 30 21:00:11.060683 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:11.060655    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-5xzm6"]
Oct 30 21:00:11.080743 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:11.080713    4521 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-c4b379-openshift-default-dockercfg-wchc9\""
Oct 30 21:00:11.080743 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:11.080738    4521 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-65968b-openshift-default-dockercfg-vzgwf\""
Oct 30 21:00:11.104645 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:11.104617    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-6ggs9"
Oct 30 21:00:11.162414 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:11.162098    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbdd2291-639e-4c1e-986d-425c4ab63dbc-istio-data\") pod \"e2e-worker-w1-c4b379-openshift-default-57fdd6d787-lp6sk\" (UID: \"fbdd2291-639e-4c1e-986d-425c4ab63dbc\") " pod="openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-lp6sk"
Oct 30 21:00:11.162414 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:11.162188    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbdd2291-639e-4c1e-986d-425c4ab63dbc-credential-socket\") pod \"e2e-worker-w1-c4b379-openshift-default-57fdd6d787-lp6sk\" (UID: \"fbdd2291-639e-4c1e-986d-425c4ab63dbc\") " pod="openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-lp6sk"
Oct 30 21:00:11.162414 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:11.162220    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/308f74fc-a395-4ad2-8902-8ed077ab9d0d-credential-socket\") pod \"e2e-worker-w5-65968b-openshift-default-68896fcddc-5xzm6\" (UID: \"308f74fc-a395-4ad2-8902-8ed077ab9d0d\") " pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-5xzm6"
Oct 30 21:00:11.162414 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:11.162244    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6mh4\" (UniqueName: \"kubernetes.io/projected/308f74fc-a395-4ad2-8902-8ed077ab9d0d-kube-api-access-p6mh4\") pod \"e2e-worker-w5-65968b-openshift-default-68896fcddc-5xzm6\" (UID: \"308f74fc-a395-4ad2-8902-8ed077ab9d0d\") " pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-5xzm6"
Oct 30 21:00:11.162414 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:11.162270    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/308f74fc-a395-4ad2-8902-8ed077ab9d0d-workload-socket\") pod \"e2e-worker-w5-65968b-openshift-default-68896fcddc-5xzm6\" (UID: \"308f74fc-a395-4ad2-8902-8ed077ab9d0d\") " pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-5xzm6"
Oct 30 21:00:11.162414 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:11.162294    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/308f74fc-a395-4ad2-8902-8ed077ab9d0d-istio-envoy\") pod \"e2e-worker-w5-65968b-openshift-default-68896fcddc-5xzm6\" (UID: \"308f74fc-a395-4ad2-8902-8ed077ab9d0d\") " pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-5xzm6"
Oct 30 21:00:11.162414 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:11.162368    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbdd2291-639e-4c1e-986d-425c4ab63dbc-workload-socket\") pod \"e2e-worker-w1-c4b379-openshift-default-57fdd6d787-lp6sk\" (UID: \"fbdd2291-639e-4c1e-986d-425c4ab63dbc\") " pod="openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-lp6sk"
Oct 30 21:00:11.162414 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:11.162395    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/308f74fc-a395-4ad2-8902-8ed077ab9d0d-workload-certs\") pod \"e2e-worker-w5-65968b-openshift-default-68896fcddc-5xzm6\" (UID: \"308f74fc-a395-4ad2-8902-8ed077ab9d0d\") " pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-5xzm6"
Oct 30 21:00:11.162414 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:11.162423    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbdd2291-639e-4c1e-986d-425c4ab63dbc-workload-certs\") pod \"e2e-worker-w1-c4b379-openshift-default-57fdd6d787-lp6sk\" (UID: \"fbdd2291-639e-4c1e-986d-425c4ab63dbc\") " pod="openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-lp6sk"
Oct 30 21:00:11.162815 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:11.162475    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2svb\" (UniqueName: \"kubernetes.io/projected/fbdd2291-639e-4c1e-986d-425c4ab63dbc-kube-api-access-w2svb\") pod \"e2e-worker-w1-c4b379-openshift-default-57fdd6d787-lp6sk\" (UID: \"fbdd2291-639e-4c1e-986d-425c4ab63dbc\") " pod="openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-lp6sk"
Oct 30 21:00:11.162815 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:11.162528    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbdd2291-639e-4c1e-986d-425c4ab63dbc-istio-envoy\") pod \"e2e-worker-w1-c4b379-openshift-default-57fdd6d787-lp6sk\" (UID: \"fbdd2291-639e-4c1e-986d-425c4ab63dbc\") " pod="openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-lp6sk"
Oct 30 21:00:11.162815 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:11.162560    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/308f74fc-a395-4ad2-8902-8ed077ab9d0d-istio-data\") pod \"e2e-worker-w5-65968b-openshift-default-68896fcddc-5xzm6\" (UID: \"308f74fc-a395-4ad2-8902-8ed077ab9d0d\") " pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-5xzm6"
Oct 30 21:00:11.162815 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:11.162620    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbdd2291-639e-4c1e-986d-425c4ab63dbc-istio-token\") pod \"e2e-worker-w1-c4b379-openshift-default-57fdd6d787-lp6sk\" (UID: \"fbdd2291-639e-4c1e-986d-425c4ab63dbc\") " pod="openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-lp6sk"
Oct 30 21:00:11.162815 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:11.162658    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbdd2291-639e-4c1e-986d-425c4ab63dbc-istio-podinfo\") pod \"e2e-worker-w1-c4b379-openshift-default-57fdd6d787-lp6sk\" (UID: \"fbdd2291-639e-4c1e-986d-425c4ab63dbc\") " pod="openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-lp6sk"
Oct 30 21:00:11.162815 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:11.162702    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/308f74fc-a395-4ad2-8902-8ed077ab9d0d-istio-podinfo\") pod \"e2e-worker-w5-65968b-openshift-default-68896fcddc-5xzm6\" (UID: \"308f74fc-a395-4ad2-8902-8ed077ab9d0d\") " pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-5xzm6"
Oct 30 21:00:11.162815 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:11.162731    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbdd2291-639e-4c1e-986d-425c4ab63dbc-istiod-ca-cert\") pod \"e2e-worker-w1-c4b379-openshift-default-57fdd6d787-lp6sk\" (UID: \"fbdd2291-639e-4c1e-986d-425c4ab63dbc\") " pod="openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-lp6sk"
Oct 30 21:00:11.162815 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:11.162769    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/308f74fc-a395-4ad2-8902-8ed077ab9d0d-istio-token\") pod \"e2e-worker-w5-65968b-openshift-default-68896fcddc-5xzm6\" (UID: \"308f74fc-a395-4ad2-8902-8ed077ab9d0d\") " pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-5xzm6"
Oct 30 21:00:11.162815 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:11.162805    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/308f74fc-a395-4ad2-8902-8ed077ab9d0d-istiod-ca-cert\") pod \"e2e-worker-w5-65968b-openshift-default-68896fcddc-5xzm6\" (UID: \"308f74fc-a395-4ad2-8902-8ed077ab9d0d\") " pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-5xzm6"
Oct 30 21:00:11.165490 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:11.165466    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-tqd8s"]
Oct 30 21:00:11.173987 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:11.173951    4521 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:00:11.174076 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:11.174039    4521 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:00:11.174114 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:11.174082    4521 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:00:11.263415 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:11.263382    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/308f74fc-a395-4ad2-8902-8ed077ab9d0d-istio-podinfo\") pod \"e2e-worker-w5-65968b-openshift-default-68896fcddc-5xzm6\" (UID: \"308f74fc-a395-4ad2-8902-8ed077ab9d0d\") " pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-5xzm6"
Oct 30 21:00:11.263573 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:11.263427    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbdd2291-639e-4c1e-986d-425c4ab63dbc-istiod-ca-cert\") pod \"e2e-worker-w1-c4b379-openshift-default-57fdd6d787-lp6sk\" (UID: \"fbdd2291-639e-4c1e-986d-425c4ab63dbc\") " pod="openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-lp6sk"
Oct 30 21:00:11.263573 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:11.263459    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/308f74fc-a395-4ad2-8902-8ed077ab9d0d-istio-token\") pod \"e2e-worker-w5-65968b-openshift-default-68896fcddc-5xzm6\" (UID: \"308f74fc-a395-4ad2-8902-8ed077ab9d0d\") " pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-5xzm6"
Oct 30 21:00:11.263573 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:11.263485    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/308f74fc-a395-4ad2-8902-8ed077ab9d0d-istiod-ca-cert\") pod \"e2e-worker-w5-65968b-openshift-default-68896fcddc-5xzm6\" (UID: \"308f74fc-a395-4ad2-8902-8ed077ab9d0d\") " pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-5xzm6"
Oct 30 21:00:11.263573 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:11.263532    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbdd2291-639e-4c1e-986d-425c4ab63dbc-istio-data\") pod \"e2e-worker-w1-c4b379-openshift-default-57fdd6d787-lp6sk\" (UID: \"fbdd2291-639e-4c1e-986d-425c4ab63dbc\") " pod="openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-lp6sk"
Oct 30 21:00:11.263807 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:11.263602    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbdd2291-639e-4c1e-986d-425c4ab63dbc-credential-socket\") pod \"e2e-worker-w1-c4b379-openshift-default-57fdd6d787-lp6sk\" (UID: \"fbdd2291-639e-4c1e-986d-425c4ab63dbc\") " pod="openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-lp6sk"
Oct 30 21:00:11.263807 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:11.263628    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/308f74fc-a395-4ad2-8902-8ed077ab9d0d-credential-socket\") pod \"e2e-worker-w5-65968b-openshift-default-68896fcddc-5xzm6\" (UID: \"308f74fc-a395-4ad2-8902-8ed077ab9d0d\") " pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-5xzm6"
Oct 30 21:00:11.263807 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:11.263650    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6mh4\" (UniqueName: \"kubernetes.io/projected/308f74fc-a395-4ad2-8902-8ed077ab9d0d-kube-api-access-p6mh4\") pod \"e2e-worker-w5-65968b-openshift-default-68896fcddc-5xzm6\" (UID: \"308f74fc-a395-4ad2-8902-8ed077ab9d0d\") " pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-5xzm6"
Oct 30 21:00:11.263807 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:11.263676    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/308f74fc-a395-4ad2-8902-8ed077ab9d0d-workload-socket\") pod \"e2e-worker-w5-65968b-openshift-default-68896fcddc-5xzm6\" (UID: \"308f74fc-a395-4ad2-8902-8ed077ab9d0d\") " pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-5xzm6"
Oct 30 21:00:11.263807 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:11.263698    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/308f74fc-a395-4ad2-8902-8ed077ab9d0d-istio-envoy\") pod \"e2e-worker-w5-65968b-openshift-default-68896fcddc-5xzm6\" (UID: \"308f74fc-a395-4ad2-8902-8ed077ab9d0d\") " pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-5xzm6"
Oct 30 21:00:11.263807 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:11.263735    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbdd2291-639e-4c1e-986d-425c4ab63dbc-workload-socket\") pod \"e2e-worker-w1-c4b379-openshift-default-57fdd6d787-lp6sk\" (UID: \"fbdd2291-639e-4c1e-986d-425c4ab63dbc\") " pod="openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-lp6sk"
Oct 30 21:00:11.263807 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:11.263758    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/308f74fc-a395-4ad2-8902-8ed077ab9d0d-workload-certs\") pod \"e2e-worker-w5-65968b-openshift-default-68896fcddc-5xzm6\" (UID: \"308f74fc-a395-4ad2-8902-8ed077ab9d0d\") " pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-5xzm6"
Oct 30 21:00:11.264181 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:11.263810    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbdd2291-639e-4c1e-986d-425c4ab63dbc-workload-certs\") pod \"e2e-worker-w1-c4b379-openshift-default-57fdd6d787-lp6sk\" (UID: \"fbdd2291-639e-4c1e-986d-425c4ab63dbc\") " pod="openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-lp6sk"
Oct 30 21:00:11.264181 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:11.263872    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2svb\" (UniqueName: \"kubernetes.io/projected/fbdd2291-639e-4c1e-986d-425c4ab63dbc-kube-api-access-w2svb\") pod \"e2e-worker-w1-c4b379-openshift-default-57fdd6d787-lp6sk\" (UID: \"fbdd2291-639e-4c1e-986d-425c4ab63dbc\") " pod="openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-lp6sk"
Oct 30 21:00:11.264181 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:11.263933    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbdd2291-639e-4c1e-986d-425c4ab63dbc-istio-envoy\") pod \"e2e-worker-w1-c4b379-openshift-default-57fdd6d787-lp6sk\" (UID: \"fbdd2291-639e-4c1e-986d-425c4ab63dbc\") " pod="openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-lp6sk"
Oct 30 21:00:11.264181 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:11.263966    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/308f74fc-a395-4ad2-8902-8ed077ab9d0d-istio-data\") pod \"e2e-worker-w5-65968b-openshift-default-68896fcddc-5xzm6\" (UID: \"308f74fc-a395-4ad2-8902-8ed077ab9d0d\") " pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-5xzm6"
Oct 30 21:00:11.264181 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:11.264033    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbdd2291-639e-4c1e-986d-425c4ab63dbc-istio-token\") pod \"e2e-worker-w1-c4b379-openshift-default-57fdd6d787-lp6sk\" (UID: \"fbdd2291-639e-4c1e-986d-425c4ab63dbc\") " pod="openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-lp6sk"
Oct 30 21:00:11.264181 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:11.264031    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbdd2291-639e-4c1e-986d-425c4ab63dbc-istio-data\") pod \"e2e-worker-w1-c4b379-openshift-default-57fdd6d787-lp6sk\" (UID: \"fbdd2291-639e-4c1e-986d-425c4ab63dbc\") " pod="openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-lp6sk"
Oct 30 21:00:11.264181 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:11.264080    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbdd2291-639e-4c1e-986d-425c4ab63dbc-istio-podinfo\") pod \"e2e-worker-w1-c4b379-openshift-default-57fdd6d787-lp6sk\" (UID: \"fbdd2291-639e-4c1e-986d-425c4ab63dbc\") " pod="openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-lp6sk"
Oct 30 21:00:11.264725 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:11.264208    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbdd2291-639e-4c1e-986d-425c4ab63dbc-credential-socket\") pod \"e2e-worker-w1-c4b379-openshift-default-57fdd6d787-lp6sk\" (UID: \"fbdd2291-639e-4c1e-986d-425c4ab63dbc\") " pod="openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-lp6sk"
Oct 30 21:00:11.264725 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:11.264286    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbdd2291-639e-4c1e-986d-425c4ab63dbc-istiod-ca-cert\") pod \"e2e-worker-w1-c4b379-openshift-default-57fdd6d787-lp6sk\" (UID: \"fbdd2291-639e-4c1e-986d-425c4ab63dbc\") " pod="openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-lp6sk"
Oct 30 21:00:11.264725 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:11.264335    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/308f74fc-a395-4ad2-8902-8ed077ab9d0d-istiod-ca-cert\") pod \"e2e-worker-w5-65968b-openshift-default-68896fcddc-5xzm6\" (UID: \"308f74fc-a395-4ad2-8902-8ed077ab9d0d\") " pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-5xzm6"
Oct 30 21:00:11.264725 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:11.264434    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/308f74fc-a395-4ad2-8902-8ed077ab9d0d-credential-socket\") pod \"e2e-worker-w5-65968b-openshift-default-68896fcddc-5xzm6\" (UID: \"308f74fc-a395-4ad2-8902-8ed077ab9d0d\") " pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-5xzm6"
Oct 30 21:00:11.264725 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:11.264495    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbdd2291-639e-4c1e-986d-425c4ab63dbc-workload-certs\") pod \"e2e-worker-w1-c4b379-openshift-default-57fdd6d787-lp6sk\" (UID: \"fbdd2291-639e-4c1e-986d-425c4ab63dbc\") " pod="openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-lp6sk"
Oct 30 21:00:11.264725 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:11.264660    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/308f74fc-a395-4ad2-8902-8ed077ab9d0d-istio-data\") pod \"e2e-worker-w5-65968b-openshift-default-68896fcddc-5xzm6\" (UID: \"308f74fc-a395-4ad2-8902-8ed077ab9d0d\") " pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-5xzm6"
Oct 30 21:00:11.264999 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:11.264763    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/308f74fc-a395-4ad2-8902-8ed077ab9d0d-workload-socket\") pod \"e2e-worker-w5-65968b-openshift-default-68896fcddc-5xzm6\" (UID: \"308f74fc-a395-4ad2-8902-8ed077ab9d0d\") " pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-5xzm6"
Oct 30 21:00:11.264999 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:11.264849    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbdd2291-639e-4c1e-986d-425c4ab63dbc-workload-socket\") pod \"e2e-worker-w1-c4b379-openshift-default-57fdd6d787-lp6sk\" (UID: \"fbdd2291-639e-4c1e-986d-425c4ab63dbc\") " pod="openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-lp6sk"
Oct 30 21:00:11.264999 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:11.264852    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/308f74fc-a395-4ad2-8902-8ed077ab9d0d-workload-certs\") pod \"e2e-worker-w5-65968b-openshift-default-68896fcddc-5xzm6\" (UID: \"308f74fc-a395-4ad2-8902-8ed077ab9d0d\") " pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-5xzm6"
Oct 30 21:00:11.266768 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:11.266723    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/308f74fc-a395-4ad2-8902-8ed077ab9d0d-istio-envoy\") pod \"e2e-worker-w5-65968b-openshift-default-68896fcddc-5xzm6\" (UID: \"308f74fc-a395-4ad2-8902-8ed077ab9d0d\") " pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-5xzm6"
Oct 30 21:00:11.266944 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:11.266927    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbdd2291-639e-4c1e-986d-425c4ab63dbc-istio-envoy\") pod \"e2e-worker-w1-c4b379-openshift-default-57fdd6d787-lp6sk\" (UID: \"fbdd2291-639e-4c1e-986d-425c4ab63dbc\") " pod="openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-lp6sk"
Oct 30 21:00:11.267012 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:11.266943    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbdd2291-639e-4c1e-986d-425c4ab63dbc-istio-podinfo\") pod \"e2e-worker-w1-c4b379-openshift-default-57fdd6d787-lp6sk\" (UID: \"fbdd2291-639e-4c1e-986d-425c4ab63dbc\") " pod="openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-lp6sk"
Oct 30 21:00:11.267060 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:11.267023    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/308f74fc-a395-4ad2-8902-8ed077ab9d0d-istio-podinfo\") pod \"e2e-worker-w5-65968b-openshift-default-68896fcddc-5xzm6\" (UID: \"308f74fc-a395-4ad2-8902-8ed077ab9d0d\") " pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-5xzm6"
Oct 30 21:00:11.282234 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:11.281394    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbdd2291-639e-4c1e-986d-425c4ab63dbc-istio-token\") pod \"e2e-worker-w1-c4b379-openshift-default-57fdd6d787-lp6sk\" (UID: \"fbdd2291-639e-4c1e-986d-425c4ab63dbc\") " pod="openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-lp6sk"
Oct 30 21:00:11.282234 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:11.281650    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2svb\" (UniqueName: \"kubernetes.io/projected/fbdd2291-639e-4c1e-986d-425c4ab63dbc-kube-api-access-w2svb\") pod \"e2e-worker-w1-c4b379-openshift-default-57fdd6d787-lp6sk\" (UID: \"fbdd2291-639e-4c1e-986d-425c4ab63dbc\") " pod="openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-lp6sk"
Oct 30 21:00:11.282234 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:11.281752    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6mh4\" (UniqueName: \"kubernetes.io/projected/308f74fc-a395-4ad2-8902-8ed077ab9d0d-kube-api-access-p6mh4\") pod \"e2e-worker-w5-65968b-openshift-default-68896fcddc-5xzm6\" (UID: \"308f74fc-a395-4ad2-8902-8ed077ab9d0d\") " pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-5xzm6"
Oct 30 21:00:11.282234 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:11.282101    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/308f74fc-a395-4ad2-8902-8ed077ab9d0d-istio-token\") pod \"e2e-worker-w5-65968b-openshift-default-68896fcddc-5xzm6\" (UID: \"308f74fc-a395-4ad2-8902-8ed077ab9d0d\") " pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-5xzm6"
Oct 30 21:00:11.340104 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:11.340002    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-lp6sk"
Oct 30 21:00:11.372686 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:11.372652    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-5xzm6"
Oct 30 21:00:11.402250 ip-10-0-131-125 kubenswrapper[4521]: W1030 21:00:11.401365    4521 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod795d7e19_cf6e_4863_bf80_28ae03af6a98.slice/crio-36f90eb210934c1392c67db8ba77f05d87b19e7f7edc61f966e956794558ee49 WatchSource:0}: Error finding container 36f90eb210934c1392c67db8ba77f05d87b19e7f7edc61f966e956794558ee49: Status 404 returned error can't find the container with id 36f90eb210934c1392c67db8ba77f05d87b19e7f7edc61f966e956794558ee49
Oct 30 21:00:11.402401 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:11.402261    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-6ggs9"]
Oct 30 21:00:11.407450 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:11.407230    4521 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:00:11.408105 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:11.407642    4521 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:00:11.408105 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:11.407711    4521 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:00:11.661343 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:11.661296    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-lp6sk"]
Oct 30 21:00:11.662730 ip-10-0-131-125 kubenswrapper[4521]: W1030 21:00:11.662691    4521 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbdd2291_639e_4c1e_986d_425c4ab63dbc.slice/crio-8fc7a41837386490c3daa999639e67973d6e6d173667bd3fc497679dd56a615a WatchSource:0}: Error finding container 8fc7a41837386490c3daa999639e67973d6e6d173667bd3fc497679dd56a615a: Status 404 returned error can't find the container with id 8fc7a41837386490c3daa999639e67973d6e6d173667bd3fc497679dd56a615a
Oct 30 21:00:11.666354 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:11.666308    4521 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:00:11.666446 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:11.666391    4521 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:00:11.666446 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:11.666418    4521 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:00:11.711778 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:11.711751    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-5xzm6"]
Oct 30 21:00:11.724204 ip-10-0-131-125 kubenswrapper[4521]: W1030 21:00:11.724179    4521 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod308f74fc_a395_4ad2_8902_8ed077ab9d0d.slice/crio-41eb1e7c07f2a5aaee86262aed569817507742dc9db250aa90e7946a2278f6f0 WatchSource:0}: Error finding container 41eb1e7c07f2a5aaee86262aed569817507742dc9db250aa90e7946a2278f6f0: Status 404 returned error can't find the container with id 41eb1e7c07f2a5aaee86262aed569817507742dc9db250aa90e7946a2278f6f0
Oct 30 21:00:11.726833 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:11.726801    4521 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:00:11.726931 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:11.726867    4521 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:00:11.726931 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:11.726898    4521 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:00:11.843841 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:11.843447    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-lp6sk" event={"ID":"fbdd2291-639e-4c1e-986d-425c4ab63dbc","Type":"ContainerStarted","Data":"026db3fbe3f5478b701e10343c4817d9fe47d7df58e5baf741b24758eb686213"}
Oct 30 21:00:11.843841 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:11.843513    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-lp6sk" event={"ID":"fbdd2291-639e-4c1e-986d-425c4ab63dbc","Type":"ContainerStarted","Data":"8fc7a41837386490c3daa999639e67973d6e6d173667bd3fc497679dd56a615a"}
Oct 30 21:00:11.847254 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:11.847223    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-5xzm6" event={"ID":"308f74fc-a395-4ad2-8902-8ed077ab9d0d","Type":"ContainerStarted","Data":"41eb1e7c07f2a5aaee86262aed569817507742dc9db250aa90e7946a2278f6f0"}
Oct 30 21:00:11.849095 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:11.849022    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-6ggs9" event={"ID":"795d7e19-cf6e-4863-bf80-28ae03af6a98","Type":"ContainerStarted","Data":"8755bbaa725db4ee5c09405b8a0c9145492ebcfceba41da023778dd0787cdb8a"}
Oct 30 21:00:11.849095 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:11.849066    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-6ggs9" event={"ID":"795d7e19-cf6e-4863-bf80-28ae03af6a98","Type":"ContainerStarted","Data":"36f90eb210934c1392c67db8ba77f05d87b19e7f7edc61f966e956794558ee49"}
Oct 30 21:00:11.852908 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:11.852886    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-tqd8s" event={"ID":"00b6adb9-3733-4ea4-9a26-f3b29991001a","Type":"ContainerStarted","Data":"e361ad15db1741dc7fcce5c213ec24d7f516d1ca35e3e8358688e2c51d1e3813"}
Oct 30 21:00:11.853079 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:11.853062    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-tqd8s" event={"ID":"00b6adb9-3733-4ea4-9a26-f3b29991001a","Type":"ContainerStarted","Data":"bb33ed467995ec4b4dc448e740cf95ca7cc41a8880caa5ddc13f3763b923e2a0"}
Oct 30 21:00:11.893402 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:11.876112    4521 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-tqd8s"
Oct 30 21:00:11.893402 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:11.877797    4521 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-lp6sk" podStartSLOduration=1.877780403 podStartE2EDuration="1.877780403s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:00:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:00:11.871935998 +0000 UTC m=+1279.217557265" watchObservedRunningTime="2026-10-30 21:00:11.877780403 +0000 UTC m=+1279.223401642"
Oct 30 21:00:11.900905 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:11.900856    4521 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-tqd8s" podStartSLOduration=1.900840715 podStartE2EDuration="1.900840715s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:00:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:00:11.899767851 +0000 UTC m=+1279.245389090" watchObservedRunningTime="2026-10-30 21:00:11.900840715 +0000 UTC m=+1279.246461958"
Oct 30 21:00:11.974637 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:11.973444    4521 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-6ggs9" podStartSLOduration=1.97342044 podStartE2EDuration="1.97342044s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:00:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:00:11.946878589 +0000 UTC m=+1279.292499829" watchObservedRunningTime="2026-10-30 21:00:11.97342044 +0000 UTC m=+1279.319041679"
Oct 30 21:00:12.106137 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:12.106103    4521 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-6ggs9"
Oct 30 21:00:12.106898 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:12.106869    4521 patch_prober.go:29] interesting pod/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-6ggs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 30 21:00:12.107029 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:12.106916    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-6ggs9" podUID="795d7e19-cf6e-4863-bf80-28ae03af6a98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 30 21:00:12.345376 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:12.345264    4521 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-lp6sk"
Oct 30 21:00:12.345722 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:12.345696    4521 patch_prober.go:29] interesting pod/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-lp6sk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 30 21:00:12.345908 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:12.345880    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-lp6sk" podUID="fbdd2291-639e-4c1e-986d-425c4ab63dbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 30 21:00:12.782426 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:12.782383    4521 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2762bcd6-4j45v"]
Oct 30 21:00:13.194516 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:13.189724    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4149346b-deed-426d-9b80-e47822d8b731-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2762bcd6-4j45v\" (UID: \"4149346b-deed-426d-9b80-e47822d8b731\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2762bcd6-4j45v"
Oct 30 21:00:13.194516 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:13.189844    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spp5c\" (UniqueName: \"kubernetes.io/projected/4149346b-deed-426d-9b80-e47822d8b731-kube-api-access-spp5c\") pod \"maas-api-revoke-sub-2762bcd6-4j45v\" (UID: \"4149346b-deed-426d-9b80-e47822d8b731\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2762bcd6-4j45v"
Oct 30 21:00:13.194516 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:13.189985    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4149346b-deed-426d-9b80-e47822d8b731-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2762bcd6-4j45v\" (UID: \"4149346b-deed-426d-9b80-e47822d8b731\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2762bcd6-4j45v"
Oct 30 21:00:13.220388 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:13.216461    4521 patch_prober.go:29] interesting pod/e2e-worker-w2-32a969-openshift-default-8945d87fc-tqd8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 21:00:13.220388 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:13.216560    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-tqd8s" podUID="00b6adb9-3733-4ea4-9a26-f3b29991001a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 21:00:13.258361 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:13.257777    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2762bcd6-4j45v"
Oct 30 21:00:13.296150 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:13.296042    4521 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:00:13.342403 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:13.316751    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4149346b-deed-426d-9b80-e47822d8b731-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2762bcd6-4j45v\" (UID: \"4149346b-deed-426d-9b80-e47822d8b731\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2762bcd6-4j45v"
Oct 30 21:00:13.342403 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:13.316848    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spp5c\" (UniqueName: \"kubernetes.io/projected/4149346b-deed-426d-9b80-e47822d8b731-kube-api-access-spp5c\") pod \"maas-api-revoke-sub-2762bcd6-4j45v\" (UID: \"4149346b-deed-426d-9b80-e47822d8b731\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2762bcd6-4j45v"
Oct 30 21:00:13.342403 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:13.316964    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4149346b-deed-426d-9b80-e47822d8b731-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2762bcd6-4j45v\" (UID: \"4149346b-deed-426d-9b80-e47822d8b731\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2762bcd6-4j45v"
Oct 30 21:00:13.342403 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:13.322309    4521 patch_prober.go:29] interesting pod/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-6ggs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 30 21:00:13.342403 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:13.322400    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-6ggs9" podUID="795d7e19-cf6e-4863-bf80-28ae03af6a98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 30 21:00:13.355539 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:13.348779    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4149346b-deed-426d-9b80-e47822d8b731-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2762bcd6-4j45v\" (UID: \"4149346b-deed-426d-9b80-e47822d8b731\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2762bcd6-4j45v"
Oct 30 21:00:13.355539 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:13.354204    4521 patch_prober.go:29] interesting pod/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-lp6sk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 30 21:00:13.355539 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:13.354285    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-lp6sk" podUID="fbdd2291-639e-4c1e-986d-425c4ab63dbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 30 21:00:13.374018 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:13.373972    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4149346b-deed-426d-9b80-e47822d8b731-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2762bcd6-4j45v\" (UID: \"4149346b-deed-426d-9b80-e47822d8b731\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2762bcd6-4j45v"
Oct 30 21:00:13.503121 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:13.503044    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spp5c\" (UniqueName: \"kubernetes.io/projected/4149346b-deed-426d-9b80-e47822d8b731-kube-api-access-spp5c\") pod \"maas-api-revoke-sub-2762bcd6-4j45v\" (UID: \"4149346b-deed-426d-9b80-e47822d8b731\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2762bcd6-4j45v"
Oct 30 21:00:13.508821 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:13.508792    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2762bcd6-4j45v"]
Oct 30 21:00:13.517161 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:13.517125    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-5xzm6" event={"ID":"308f74fc-a395-4ad2-8902-8ed077ab9d0d","Type":"ContainerStarted","Data":"a65276986ef16a935ea5918e247501cdea333b646757020527c9f377f598a28d"}
Oct 30 21:00:13.530707 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:13.530670    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2762bcd6-4j45v"
Oct 30 21:00:13.534343 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:13.534270    4521 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-5xzm6" podStartSLOduration=2.534250687 podStartE2EDuration="2.534250687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:00:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:00:13.507841818 +0000 UTC m=+1280.853463061" watchObservedRunningTime="2026-10-30 21:00:13.534250687 +0000 UTC m=+1280.879871928"
Oct 30 21:00:13.871810 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:13.871716    4521 patch_prober.go:29] interesting pod/e2e-worker-w2-32a969-openshift-default-8945d87fc-tqd8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 21:00:13.872409 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:13.872381    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-tqd8s" podUID="00b6adb9-3733-4ea4-9a26-f3b29991001a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 21:00:14.128408 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:14.120456    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-39fb96/premium-w6-39fb96-kserve-b9cf7f75c-96blx" podUID="18ca0a94-c322-41fa-9fd9-0f14419b5075" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:00:14.128408 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:14.124381    4521 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-5a8f94/distinct-w3-5a8f94-kserve-5f8b544d59-mf95j"
Oct 30 21:00:14.344572 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:14.343466    4521 patch_prober.go:29] interesting pod/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-6ggs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 30 21:00:14.344572 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:14.343534    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-6ggs9" podUID="795d7e19-cf6e-4863-bf80-28ae03af6a98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 30 21:00:14.345248 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:14.345155    4521 patch_prober.go:29] interesting pod/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-lp6sk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 30 21:00:14.345248 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:14.345209    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-lp6sk" podUID="fbdd2291-639e-4c1e-986d-425c4ab63dbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 30 21:00:14.384433 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:14.384282    4521 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-5xzm6"
Oct 30 21:00:14.494726 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:14.481451    4521 patch_prober.go:29] interesting pod/e2e-worker-w5-65968b-openshift-default-68896fcddc-5xzm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 30 21:00:14.494726 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:14.481508    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-5xzm6" podUID="308f74fc-a395-4ad2-8902-8ed077ab9d0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 30 21:00:14.494726 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:14.484053    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2762bcd6-4j45v"]
Oct 30 21:00:14.549560 ip-10-0-131-125 kubenswrapper[4521]: W1030 21:00:14.549516    4521 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4149346b_deed_426d_9b80_e47822d8b731.slice/crio-d2ddc77fee78e139cb145aaa066af36233e72466fc525d050a5f9ca2006b5678 WatchSource:0}: Error finding container d2ddc77fee78e139cb145aaa066af36233e72466fc525d050a5f9ca2006b5678: Status 404 returned error can't find the container with id d2ddc77fee78e139cb145aaa066af36233e72466fc525d050a5f9ca2006b5678
Oct 30 21:00:14.846348 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:14.804126    4521 scope.go:122] "RemoveContainer" containerID="7be1b30dd65665678dbc107591759f5cd9a166ad0356e7fae892af4552330fb3"
Oct 30 21:00:14.983599 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:14.909474    4521 patch_prober.go:29] interesting pod/e2e-worker-w2-32a969-openshift-default-8945d87fc-tqd8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 21:00:14.983599 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:14.980504    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-tqd8s" podUID="00b6adb9-3733-4ea4-9a26-f3b29991001a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 21:00:14.983599 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:14.909527    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a8f94/simulator-w3-5a8f94-kserve-77cc88cc89-n4sl6" podUID="1fa6900c-eee2-4f00-b0ec-248def48dfc4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:00:15.014947 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:15.014529    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a8f94/premium-w3-5a8f94-kserve-bb96b6848-5hk8h" podUID="24ccc37d-027a-4097-8da5-04df417aef94" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Oct 30 21:00:15.029896 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:15.029502    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-65968b/premium-w5-65968b-kserve-67977485f7-c9jpv" podUID="92b2dd65-8a83-436d-afa8-50e5f0d11fa0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:00:15.058524 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:15.058051    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c4b379/premium-w1-c4b379-kserve-575f896d49-2n6gz" podUID="2aac3e8e-d631-4b9c-bdbb-3b4b1df133a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:00:15.116429 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:15.116314    4521 patch_prober.go:29] interesting pod/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-6ggs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 30 21:00:15.116740 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:15.116400    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-6ggs9" podUID="795d7e19-cf6e-4863-bf80-28ae03af6a98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 30 21:00:15.128896 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:15.121412    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-39fb96/premium-w6-39fb96-kserve-b9cf7f75c-96blx" podUID="18ca0a94-c322-41fa-9fd9-0f14419b5075" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:00:15.128896 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:15.122186    4521 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-models-e2e-worker-w3-5a8f94/distinct-w3-5a8f94-kserve-5f8b544d59-mf95j" podUID="b8ac1cca-4ebd-474c-b32f-a3595a4cd847" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/health\": context deadline exceeded"
Oct 30 21:00:15.133532 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:15.133473    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c2d1eb/premium-w4-c2d1eb-kserve-8696fbd9b9-mxqhm" podUID="4bb0f898-b77f-419d-bff4-6413e82ef51d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:00:15.138087 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:15.138050    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-39fb96/premium-w6-39fb96-kserve-b9cf7f75c-96blx"
Oct 30 21:00:15.350909 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:15.349149    4521 patch_prober.go:29] interesting pod/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-lp6sk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 30 21:00:15.350909 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:15.349213    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-lp6sk" podUID="fbdd2291-639e-4c1e-986d-425c4ab63dbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 30 21:00:15.417757 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:15.417193    4521 patch_prober.go:29] interesting pod/e2e-worker-w5-65968b-openshift-default-68896fcddc-5xzm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 30 21:00:15.417757 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:15.417254    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-5xzm6" podUID="308f74fc-a395-4ad2-8902-8ed077ab9d0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 30 21:00:15.685147 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:15.685116    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2762bcd6-4j45v" event={"ID":"4149346b-deed-426d-9b80-e47822d8b731","Type":"ContainerStarted","Data":"d2ddc77fee78e139cb145aaa066af36233e72466fc525d050a5f9ca2006b5678"}
Oct 30 21:00:15.815965 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:15.809616    4521 scope.go:122] "RemoveContainer" containerID="b42a19e8e609608a0cf9a112877cc76091fe4dcada28ce1fb6cc83b85f77222c"
Oct 30 21:00:15.877688 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:15.877192    4521 patch_prober.go:29] interesting pod/e2e-worker-w2-32a969-openshift-default-8945d87fc-tqd8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 21:00:15.877688 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:15.877251    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-tqd8s" podUID="00b6adb9-3733-4ea4-9a26-f3b29991001a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 21:00:15.898528 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:15.898471    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-32a969/simulator-w2-32a969-kserve-6cc8dfc44c-dwhlt" podUID="62a6adf7-d17a-4fec-a707-a2d549b67753" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:00:15.929763 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:15.927430    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a0c5b/premium-w0-4a0c5b-kserve-85dcf89f57-mscdn" podUID="00e42758-77ce-4663-adfa-2202c0aae24a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:00:15.985942 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:15.985899    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a8f94/simulator-w3-5a8f94-kserve-77cc88cc89-n4sl6" podUID="1fa6900c-eee2-4f00-b0ec-248def48dfc4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:00:15.987666 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:15.986595    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5a8f94/simulator-w3-5a8f94-kserve-77cc88cc89-n4sl6"
Oct 30 21:00:16.015291 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:16.015229    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a8f94/premium-w3-5a8f94-kserve-bb96b6848-5hk8h" podUID="24ccc37d-027a-4097-8da5-04df417aef94" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Oct 30 21:00:16.015580 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:16.015553    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5a8f94/premium-w3-5a8f94-kserve-bb96b6848-5hk8h"
Oct 30 21:00:16.036045 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:16.035952    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-65968b/premium-w5-65968b-kserve-67977485f7-c9jpv" podUID="92b2dd65-8a83-436d-afa8-50e5f0d11fa0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 30 21:00:16.036725 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:16.036705    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-65968b/premium-w5-65968b-kserve-67977485f7-c9jpv"
Oct 30 21:00:16.064563 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:16.064519    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c4b379/premium-w1-c4b379-kserve-575f896d49-2n6gz" podUID="2aac3e8e-d631-4b9c-bdbb-3b4b1df133a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:00:16.065193 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:16.065173    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c4b379/premium-w1-c4b379-kserve-575f896d49-2n6gz"
Oct 30 21:00:16.136347 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:16.134023    4521 patch_prober.go:29] interesting pod/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-6ggs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 30 21:00:16.136347 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:16.134094    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-6ggs9" podUID="795d7e19-cf6e-4863-bf80-28ae03af6a98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 30 21:00:16.190361 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:16.189815    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c2d1eb/premium-w4-c2d1eb-kserve-8696fbd9b9-mxqhm" podUID="4bb0f898-b77f-419d-bff4-6413e82ef51d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded"
Oct 30 21:00:16.306918 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:16.278727    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-39fb96/premium-w6-39fb96-kserve-b9cf7f75c-96blx" podUID="18ca0a94-c322-41fa-9fd9-0f14419b5075" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded"
Oct 30 21:00:16.352367 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:16.350011    4521 patch_prober.go:29] interesting pod/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-lp6sk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 30 21:00:16.352367 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:16.350074    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-lp6sk" podUID="fbdd2291-639e-4c1e-986d-425c4ab63dbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 30 21:00:16.385355 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:16.384415    4521 patch_prober.go:29] interesting pod/e2e-worker-w5-65968b-openshift-default-68896fcddc-5xzm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 30 21:00:16.385355 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:16.384476    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-5xzm6" podUID="308f74fc-a395-4ad2-8902-8ed077ab9d0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 30 21:00:16.728347 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:16.726819    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c2d1eb/premium-w4-c2d1eb-kserve-8696fbd9b9-mxqhm"
Oct 30 21:00:16.840353 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:16.826969    4521 scope.go:122] "RemoveContainer" containerID="8354941073a53fcc812e0428dd49e48225c40acf3f7f36e0aefca53a4060fbbe"
Oct 30 21:00:16.870188 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:16.870099    4521 patch_prober.go:29] interesting pod/e2e-worker-w2-32a969-openshift-default-8945d87fc-tqd8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 21:00:16.876175 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:16.875832    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-tqd8s" podUID="00b6adb9-3733-4ea4-9a26-f3b29991001a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 21:00:16.928857 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:16.901956    4521 generic.go:356] "Generic (PLEG): container finished" podID="49b3ef08-eca0-4a62-885d-bcf8fc23cb6d" containerID="422ee52c4e3c510f701d628ad19779c76f29147db875f2a52d18ca208a037982" exitCode=60
Oct 30 21:00:16.928857 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:16.902257    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-39fb96-29889900-sbp6w" event={"ID":"49b3ef08-eca0-4a62-885d-bcf8fc23cb6d","Type":"ContainerDied","Data":"422ee52c4e3c510f701d628ad19779c76f29147db875f2a52d18ca208a037982"}
Oct 30 21:00:16.928857 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:16.902348    4521 scope.go:122] "RemoveContainer" containerID="7be1b30dd65665678dbc107591759f5cd9a166ad0356e7fae892af4552330fb3"
Oct 30 21:00:16.928857 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:16.902939    4521 scope.go:122] "RemoveContainer" containerID="422ee52c4e3c510f701d628ad19779c76f29147db875f2a52d18ca208a037982"
Oct 30 21:00:16.928857 ip-10-0-131-125 kubenswrapper[4521]: E1030 21:00:16.903445    4521 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-39fb96-29889900-sbp6w_odh-ai-gateway-infra(49b3ef08-eca0-4a62-885d-bcf8fc23cb6d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-39fb96-29889900-sbp6w" podUID="49b3ef08-eca0-4a62-885d-bcf8fc23cb6d"
Oct 30 21:00:16.973797 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:16.967638    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-39fb96/premium-w6-39fb96-kserve-b9cf7f75c-96blx"
Oct 30 21:00:16.973797 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:16.968182    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a0c5b/premium-w0-4a0c5b-kserve-85dcf89f57-mscdn" podUID="00e42758-77ce-4663-adfa-2202c0aae24a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Oct 30 21:00:16.973797 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:16.969172    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-32a969/simulator-w2-32a969-kserve-6cc8dfc44c-dwhlt" podUID="62a6adf7-d17a-4fec-a707-a2d549b67753" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:00:16.974742 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:16.974671    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4a0c5b/premium-w0-4a0c5b-kserve-85dcf89f57-mscdn"
Oct 30 21:00:16.975627 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:16.975601    4521 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-c2d1eb/premium-w4-c2d1eb-kserve-8696fbd9b9-mxqhm" podUID="4bb0f898-b77f-419d-bff4-6413e82ef51d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": context deadline exceeded"
Oct 30 21:00:16.976227 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:16.974964    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-32a969/simulator-w2-32a969-kserve-6cc8dfc44c-dwhlt"
Oct 30 21:00:16.996848 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:16.995132    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-65968b/premium-w5-65968b-kserve-67977485f7-c9jpv"
Oct 30 21:00:17.040805 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:17.034553    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a8f94/premium-w3-5a8f94-kserve-bb96b6848-5hk8h" podUID="24ccc37d-027a-4097-8da5-04df417aef94" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Oct 30 21:00:17.042200 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:17.041639    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a8f94/simulator-w3-5a8f94-kserve-77cc88cc89-n4sl6" podUID="1fa6900c-eee2-4f00-b0ec-248def48dfc4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:00:17.083386 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:17.082404    4521 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-32a969/distinct-w2-32a969-kserve-7f798b664f-jdh82"
Oct 30 21:00:17.092174 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:17.083643    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c4b379/premium-w1-c4b379-kserve-575f896d49-2n6gz" podUID="2aac3e8e-d631-4b9c-bdbb-3b4b1df133a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:00:17.120405 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:17.116571    4521 patch_prober.go:29] interesting pod/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-6ggs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 30 21:00:17.120405 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:17.116638    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-6ggs9" podUID="795d7e19-cf6e-4863-bf80-28ae03af6a98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 30 21:00:17.136453 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:17.122475    4521 generic.go:356] "Generic (PLEG): container finished" podID="4149346b-deed-426d-9b80-e47822d8b731" containerID="5ecfb1b5956a52eaf146dec11f6db14e3bba96d43a1549bd12ecfbce3da5220b" exitCode=0
Oct 30 21:00:17.136453 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:17.122649    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2762bcd6-4j45v" event={"ID":"4149346b-deed-426d-9b80-e47822d8b731","Type":"ContainerDied","Data":"5ecfb1b5956a52eaf146dec11f6db14e3bba96d43a1549bd12ecfbce3da5220b"}
Oct 30 21:00:17.197833 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:17.197777    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c2d1eb/premium-w4-c2d1eb-kserve-8696fbd9b9-mxqhm" podUID="4bb0f898-b77f-419d-bff4-6413e82ef51d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:00:17.219314 ip-10-0-131-125 kubenswrapper[4521]: E1030 21:00:17.219280    4521 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lkn2k" containerName="extractor"
Oct 30 21:00:17.344950 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:17.344914    4521 patch_prober.go:29] interesting pod/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-lp6sk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 30 21:00:17.345183 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:17.345158    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-lp6sk" podUID="fbdd2291-639e-4c1e-986d-425c4ab63dbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 30 21:00:17.408347 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:17.407717    4521 patch_prober.go:29] interesting pod/e2e-worker-w5-65968b-openshift-default-68896fcddc-5xzm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 30 21:00:17.408347 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:17.407781    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-5xzm6" podUID="308f74fc-a395-4ad2-8902-8ed077ab9d0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 30 21:00:17.499853 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:17.499824    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c2d1eb/premium-w4-c2d1eb-kserve-8696fbd9b9-mxqhm"
Oct 30 21:00:17.647430 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:17.629762    4521 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-32a969/distinct-w2-32a969-kserve-7f798b664f-jdh82"
Oct 30 21:00:17.663205 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:17.662898    4521 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-5a8f94/simulator-w3-5a8f94-kserve-77cc88cc89-n4sl6" podUID="1fa6900c-eee2-4f00-b0ec-248def48dfc4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:00:17.872363 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:17.870705    4521 patch_prober.go:29] interesting pod/e2e-worker-w2-32a969-openshift-default-8945d87fc-tqd8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 21:00:17.872363 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:17.870771    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-tqd8s" podUID="00b6adb9-3733-4ea4-9a26-f3b29991001a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 21:00:17.982348 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:17.981205    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-32a969/simulator-w2-32a969-kserve-6cc8dfc44c-dwhlt" podUID="62a6adf7-d17a-4fec-a707-a2d549b67753" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Oct 30 21:00:17.982348 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:17.981523    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a0c5b/premium-w0-4a0c5b-kserve-85dcf89f57-mscdn" podUID="00e42758-77ce-4663-adfa-2202c0aae24a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:00:18.040350 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:18.038951    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a8f94/premium-w3-5a8f94-kserve-bb96b6848-5hk8h" podUID="24ccc37d-027a-4097-8da5-04df417aef94" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:00:18.042144 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:18.042104    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a8f94/simulator-w3-5a8f94-kserve-77cc88cc89-n4sl6" podUID="1fa6900c-eee2-4f00-b0ec-248def48dfc4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:00:18.084509 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:18.084451    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c4b379/premium-w1-c4b379-kserve-575f896d49-2n6gz" podUID="2aac3e8e-d631-4b9c-bdbb-3b4b1df133a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:00:18.085499 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:18.085466    4521 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-39fb96-29889900-sbp6w"]
Oct 30 21:00:18.108267 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:18.107819    4521 patch_prober.go:29] interesting pod/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-6ggs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 30 21:00:18.108267 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:18.107881    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-6ggs9" podUID="795d7e19-cf6e-4863-bf80-28ae03af6a98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 30 21:00:18.308277 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:18.308239    4521 generic.go:356] "Generic (PLEG): container finished" podID="f0bc9c01-1d10-4e3d-9876-f24637b968ef" containerID="27851c66fa4928bf0ec43444e747900dba7f4d758608875183cc68c7541d8ecf" exitCode=60
Oct 30 21:00:18.308712 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:18.308685    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-65968b-29889900-hkfkj" event={"ID":"f0bc9c01-1d10-4e3d-9876-f24637b968ef","Type":"ContainerDied","Data":"27851c66fa4928bf0ec43444e747900dba7f4d758608875183cc68c7541d8ecf"}
Oct 30 21:00:18.308978 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:18.308945    4521 scope.go:122] "RemoveContainer" containerID="b42a19e8e609608a0cf9a112877cc76091fe4dcada28ce1fb6cc83b85f77222c"
Oct 30 21:00:18.326746 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:18.326715    4521 scope.go:122] "RemoveContainer" containerID="27851c66fa4928bf0ec43444e747900dba7f4d758608875183cc68c7541d8ecf"
Oct 30 21:00:18.345950 ip-10-0-131-125 kubenswrapper[4521]: E1030 21:00:18.345913    4521 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-65968b-29889900-hkfkj_odh-ai-gateway-infra(f0bc9c01-1d10-4e3d-9876-f24637b968ef)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-65968b-29889900-hkfkj" podUID="f0bc9c01-1d10-4e3d-9876-f24637b968ef"
Oct 30 21:00:18.376811 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:18.376772    4521 patch_prober.go:29] interesting pod/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-lp6sk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 30 21:00:18.379942 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:18.379907    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-lp6sk" podUID="fbdd2291-639e-4c1e-986d-425c4ab63dbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 30 21:00:18.380159 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:18.378790    4521 patch_prober.go:29] interesting pod/e2e-worker-w5-65968b-openshift-default-68896fcddc-5xzm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 30 21:00:18.380278 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:18.380257    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-5xzm6" podUID="308f74fc-a395-4ad2-8902-8ed077ab9d0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 30 21:00:18.428350 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:18.423832    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c4b379-29889900-sbvxl" event={"ID":"1e319e8b-6373-4ad3-b549-086f4a0c0f25","Type":"ContainerStarted","Data":"2bb6974367a84213ca55ee650635b48cfed923074ab790f99c266c7e11892084"}
Oct 30 21:00:18.551915 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:18.551852    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c2d1eb/premium-w4-c2d1eb-kserve-8696fbd9b9-mxqhm" podUID="4bb0f898-b77f-419d-bff4-6413e82ef51d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:00:18.723214 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:18.723182    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-32a969/distinct-w2-32a969-kserve-7f798b664f-jdh82"
Oct 30 21:00:18.889819 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:18.889705    4521 patch_prober.go:29] interesting pod/e2e-worker-w2-32a969-openshift-default-8945d87fc-tqd8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 21:00:18.890085 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:18.890042    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-tqd8s" podUID="00b6adb9-3733-4ea4-9a26-f3b29991001a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 21:00:19.003349 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:18.982426    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a0c5b/premium-w0-4a0c5b-kserve-85dcf89f57-mscdn" podUID="00e42758-77ce-4663-adfa-2202c0aae24a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:00:19.003349 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:19.001628    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-32a969/simulator-w2-32a969-kserve-6cc8dfc44c-dwhlt" podUID="62a6adf7-d17a-4fec-a707-a2d549b67753" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Oct 30 21:00:19.077009 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:19.075897    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a8f94/premium-w3-5a8f94-kserve-bb96b6848-5hk8h" podUID="24ccc37d-027a-4097-8da5-04df417aef94" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:00:19.077009 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:19.076394    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a8f94/simulator-w3-5a8f94-kserve-77cc88cc89-n4sl6" podUID="1fa6900c-eee2-4f00-b0ec-248def48dfc4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:00:19.093083 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:19.092462    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c4b379/premium-w1-c4b379-kserve-575f896d49-2n6gz" podUID="2aac3e8e-d631-4b9c-bdbb-3b4b1df133a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:00:19.109348 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:19.108085    4521 patch_prober.go:29] interesting pod/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-6ggs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 30 21:00:19.109348 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:19.108156    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-6ggs9" podUID="795d7e19-cf6e-4863-bf80-28ae03af6a98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 30 21:00:19.182713 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:19.182627    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c4b379/premium-w1-c4b379-kserve-575f896d49-2n6gz"
Oct 30 21:00:19.194345 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:19.184131    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-32a969/simulator-w2-32a969-kserve-6cc8dfc44c-dwhlt"
Oct 30 21:00:19.302047 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:19.283612    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5a8f94/premium-w3-5a8f94-kserve-bb96b6848-5hk8h"
Oct 30 21:00:19.368346 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:19.368003    4521 patch_prober.go:29] interesting pod/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-lp6sk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 30 21:00:19.368346 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:19.368090    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-lp6sk" podUID="fbdd2291-639e-4c1e-986d-425c4ab63dbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 30 21:00:19.409222 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:19.408416    4521 patch_prober.go:29] interesting pod/e2e-worker-w5-65968b-openshift-default-68896fcddc-5xzm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 30 21:00:19.409222 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:19.408477    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-5xzm6" podUID="308f74fc-a395-4ad2-8902-8ed077ab9d0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 30 21:00:19.563408 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:19.561701    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c2d1eb/premium-w4-c2d1eb-kserve-8696fbd9b9-mxqhm" podUID="4bb0f898-b77f-419d-bff4-6413e82ef51d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:00:19.563408 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:19.562218    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c2d1eb/premium-w4-c2d1eb-kserve-8696fbd9b9-mxqhm"
Oct 30 21:00:20.032553 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:19.946525    4521 patch_prober.go:29] interesting pod/e2e-worker-w2-32a969-openshift-default-8945d87fc-tqd8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 21:00:20.032553 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:19.946607    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-tqd8s" podUID="00b6adb9-3733-4ea4-9a26-f3b29991001a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 21:00:20.102138 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:20.102085    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a0c5b/premium-w0-4a0c5b-kserve-85dcf89f57-mscdn" podUID="00e42758-77ce-4663-adfa-2202c0aae24a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:00:20.155290 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:20.155235    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-65968b/premium-w5-65968b-kserve-67977485f7-c9jpv" podUID="92b2dd65-8a83-436d-afa8-50e5f0d11fa0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:00:20.159354 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:20.156713    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-39fb96-29889900-sbp6w"
Oct 30 21:00:20.159354 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:20.155401    4521 patch_prober.go:29] interesting pod/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-6ggs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 30 21:00:20.159354 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:20.157517    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-6ggs9" podUID="795d7e19-cf6e-4863-bf80-28ae03af6a98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 30 21:00:20.252537 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:20.252486    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a8f94/simulator-w3-5a8f94-kserve-77cc88cc89-n4sl6" podUID="1fa6900c-eee2-4f00-b0ec-248def48dfc4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:00:20.354361 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:20.345778    4521 patch_prober.go:29] interesting pod/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-lp6sk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 30 21:00:20.354653 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:20.354616    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-lp6sk" podUID="fbdd2291-639e-4c1e-986d-425c4ab63dbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 30 21:00:20.482349 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:20.374785    4521 patch_prober.go:29] interesting pod/e2e-worker-w5-65968b-openshift-default-68896fcddc-5xzm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 30 21:00:20.482349 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:20.374867    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-5xzm6" podUID="308f74fc-a395-4ad2-8902-8ed077ab9d0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 30 21:00:20.482349 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:20.376373    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49b3ef08-eca0-4a62-885d-bcf8fc23cb6d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/49b3ef08-eca0-4a62-885d-bcf8fc23cb6d-maas-api-service-ca\") pod \"49b3ef08-eca0-4a62-885d-bcf8fc23cb6d\" (UID: \"49b3ef08-eca0-4a62-885d-bcf8fc23cb6d\") "
Oct 30 21:00:20.482349 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:20.376469    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49b3ef08-eca0-4a62-885d-bcf8fc23cb6d-kube-api-access-w8q4h\" (UniqueName: \"kubernetes.io/projected/49b3ef08-eca0-4a62-885d-bcf8fc23cb6d-kube-api-access-w8q4h\") pod \"49b3ef08-eca0-4a62-885d-bcf8fc23cb6d\" (UID: \"49b3ef08-eca0-4a62-885d-bcf8fc23cb6d\") "
Oct 30 21:00:20.482349 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:20.428628    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49b3ef08-eca0-4a62-885d-bcf8fc23cb6d-kube-api-access-w8q4h" pod "49b3ef08-eca0-4a62-885d-bcf8fc23cb6d" (UID: "49b3ef08-eca0-4a62-885d-bcf8fc23cb6d"). InnerVolumeSpecName "kube-api-access-w8q4h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:00:20.487655 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:20.487613    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49b3ef08-eca0-4a62-885d-bcf8fc23cb6d-maas-api-service-ca" pod "49b3ef08-eca0-4a62-885d-bcf8fc23cb6d" (UID: "49b3ef08-eca0-4a62-885d-bcf8fc23cb6d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:00:20.494919 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:20.493004    4521 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/49b3ef08-eca0-4a62-885d-bcf8fc23cb6d-maas-api-service-ca\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:00:20.494919 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:20.493037    4521 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8q4h\" (UniqueName: \"kubernetes.io/projected/49b3ef08-eca0-4a62-885d-bcf8fc23cb6d-kube-api-access-w8q4h\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:00:20.519356 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:20.518877    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2762bcd6-4j45v"
Oct 30 21:00:20.564863 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:20.564763    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c2d1eb/premium-w4-c2d1eb-kserve-8696fbd9b9-mxqhm" podUID="4bb0f898-b77f-419d-bff4-6413e82ef51d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:00:20.617783 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:20.617745    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4149346b-deed-426d-9b80-e47822d8b731-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4149346b-deed-426d-9b80-e47822d8b731-maas-api-xks-ca\") pod \"4149346b-deed-426d-9b80-e47822d8b731\" (UID: \"4149346b-deed-426d-9b80-e47822d8b731\") "
Oct 30 21:00:20.618087 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:20.618067    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4149346b-deed-426d-9b80-e47822d8b731-kube-api-access-spp5c\" (UniqueName: \"kubernetes.io/projected/4149346b-deed-426d-9b80-e47822d8b731-kube-api-access-spp5c\") pod \"4149346b-deed-426d-9b80-e47822d8b731\" (UID: \"4149346b-deed-426d-9b80-e47822d8b731\") "
Oct 30 21:00:20.618229 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:20.618216    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4149346b-deed-426d-9b80-e47822d8b731-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4149346b-deed-426d-9b80-e47822d8b731-maas-api-openshift-ca\") pod \"4149346b-deed-426d-9b80-e47822d8b731\" (UID: \"4149346b-deed-426d-9b80-e47822d8b731\") "
Oct 30 21:00:20.632610 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:20.632570    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4149346b-deed-426d-9b80-e47822d8b731-maas-api-openshift-ca" pod "4149346b-deed-426d-9b80-e47822d8b731" (UID: "4149346b-deed-426d-9b80-e47822d8b731"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:00:20.633476 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:20.633441    4521 generic.go:356] "Generic (PLEG): container finished" podID="1e319e8b-6373-4ad3-b549-086f4a0c0f25" containerID="2bb6974367a84213ca55ee650635b48cfed923074ab790f99c266c7e11892084" exitCode=60
Oct 30 21:00:20.635338 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:20.635297    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c4b379-29889900-sbvxl" event={"ID":"1e319e8b-6373-4ad3-b549-086f4a0c0f25","Type":"ContainerDied","Data":"2bb6974367a84213ca55ee650635b48cfed923074ab790f99c266c7e11892084"}
Oct 30 21:00:20.635558 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:20.635540    4521 scope.go:122] "RemoveContainer" containerID="8354941073a53fcc812e0428dd49e48225c40acf3f7f36e0aefca53a4060fbbe"
Oct 30 21:00:20.665350 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:20.637026    4521 scope.go:122] "RemoveContainer" containerID="2bb6974367a84213ca55ee650635b48cfed923074ab790f99c266c7e11892084"
Oct 30 21:00:20.666189 ip-10-0-131-125 kubenswrapper[4521]: E1030 21:00:20.666151    4521 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-c4b379-29889900-sbvxl_odh-ai-gateway-infra(1e319e8b-6373-4ad3-b549-086f4a0c0f25)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c4b379-29889900-sbvxl" podUID="1e319e8b-6373-4ad3-b549-086f4a0c0f25"
Oct 30 21:00:20.681352 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:20.673898    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4149346b-deed-426d-9b80-e47822d8b731-maas-api-xks-ca" pod "4149346b-deed-426d-9b80-e47822d8b731" (UID: "4149346b-deed-426d-9b80-e47822d8b731"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:00:20.789430 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:20.731567    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c2d1eb/premium-w4-c2d1eb-kserve-8696fbd9b9-mxqhm"
Oct 30 21:00:20.789840 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:20.778719    4521 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4149346b-deed-426d-9b80-e47822d8b731-maas-api-openshift-ca\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:00:20.899750 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:20.899657    4521 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4149346b-deed-426d-9b80-e47822d8b731-maas-api-xks-ca\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:00:20.913902 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:20.912685    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-32a969/distinct-w2-32a969-kserve-7f798b664f-jdh82" podUID="887f1d27-bf24-4b57-b6e4-8fb095df7a50" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:00:20.914476 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:20.912835    4521 patch_prober.go:29] interesting pod/e2e-worker-w2-32a969-openshift-default-8945d87fc-tqd8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 21:00:20.931975 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:20.931936    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-tqd8s" podUID="00b6adb9-3733-4ea4-9a26-f3b29991001a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 21:00:20.943753 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:20.940126    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4149346b-deed-426d-9b80-e47822d8b731-kube-api-access-spp5c" pod "4149346b-deed-426d-9b80-e47822d8b731" (UID: "4149346b-deed-426d-9b80-e47822d8b731"). InnerVolumeSpecName "kube-api-access-spp5c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:00:21.036730 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:21.036675    4521 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-spp5c\" (UniqueName: \"kubernetes.io/projected/4149346b-deed-426d-9b80-e47822d8b731-kube-api-access-spp5c\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:00:21.070744 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:21.070709    4521 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-65968b-29889900-hkfkj"]
Oct 30 21:00:21.103880 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:21.103837    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a0c5b/premium-w0-4a0c5b-kserve-85dcf89f57-mscdn" podUID="00e42758-77ce-4663-adfa-2202c0aae24a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:00:21.111345 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:21.105938    4521 patch_prober.go:29] interesting pod/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-6ggs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 30 21:00:21.124345 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:21.112129    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-6ggs9" podUID="795d7e19-cf6e-4863-bf80-28ae03af6a98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 30 21:00:21.127032 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:21.127002    4521 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c4b379-29889900-sbvxl"]
Oct 30 21:00:21.186639 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:21.185898    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5a8f94/simulator-w3-5a8f94-kserve-77cc88cc89-n4sl6"
Oct 30 21:00:21.186639 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:21.185992    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4a0c5b/premium-w0-4a0c5b-kserve-85dcf89f57-mscdn"
Oct 30 21:00:21.364661 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:21.364624    4521 patch_prober.go:29] interesting pod/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-lp6sk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 30 21:00:21.366049 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:21.366008    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-lp6sk" podUID="fbdd2291-639e-4c1e-986d-425c4ab63dbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 30 21:00:21.375158 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:21.375128    4521 patch_prober.go:29] interesting pod/e2e-worker-w5-65968b-openshift-default-68896fcddc-5xzm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 30 21:00:21.375400 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:21.375373    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-5xzm6" podUID="308f74fc-a395-4ad2-8902-8ed077ab9d0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 30 21:00:21.639347 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:21.628408    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-65968b-29889900-hkfkj"
Oct 30 21:00:21.656237 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:21.656197    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-39fb96-29889900-sbp6w" event={"ID":"49b3ef08-eca0-4a62-885d-bcf8fc23cb6d","Type":"ContainerDied","Data":"51f30029c986e264fe8726611967e3523e8e3d720fcd25b6fa3e6e76b92dd5c9"}
Oct 30 21:00:21.656649 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:21.656628    4521 scope.go:122] "RemoveContainer" containerID="422ee52c4e3c510f701d628ad19779c76f29147db875f2a52d18ca208a037982"
Oct 30 21:00:21.656844 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:21.656525    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-39fb96-29889900-sbp6w"
Oct 30 21:00:21.760368 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:21.756598    4521 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-39fb96-29889900-sbp6w"]
Oct 30 21:00:21.760368 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:21.756652    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-65968b-29889900-hkfkj" event={"ID":"f0bc9c01-1d10-4e3d-9876-f24637b968ef","Type":"ContainerDied","Data":"9d3099faa50f306b41d109ef0a334938792a335f053270604c2e25b0cdf32c82"}
Oct 30 21:00:21.760368 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:21.756702    4521 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-39fb96-29889900-sbp6w"]
Oct 30 21:00:21.760368 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:21.756752    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-65968b-29889900-hkfkj"
Oct 30 21:00:21.772060 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:21.762710    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2762bcd6-4j45v" event={"ID":"4149346b-deed-426d-9b80-e47822d8b731","Type":"ContainerDied","Data":"d2ddc77fee78e139cb145aaa066af36233e72466fc525d050a5f9ca2006b5678"}
Oct 30 21:00:21.772060 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:21.762765    4521 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d2ddc77fee78e139cb145aaa066af36233e72466fc525d050a5f9ca2006b5678"
Oct 30 21:00:21.772060 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:21.762828    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2762bcd6-4j45v"
Oct 30 21:00:21.783141 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:21.780280    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0bc9c01-1d10-4e3d-9876-f24637b968ef-kube-api-access-7mjmq\" (UniqueName: \"kubernetes.io/projected/f0bc9c01-1d10-4e3d-9876-f24637b968ef-kube-api-access-7mjmq\") pod \"f0bc9c01-1d10-4e3d-9876-f24637b968ef\" (UID: \"f0bc9c01-1d10-4e3d-9876-f24637b968ef\") "
Oct 30 21:00:21.783141 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:21.782427    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0bc9c01-1d10-4e3d-9876-f24637b968ef-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0bc9c01-1d10-4e3d-9876-f24637b968ef-maas-api-service-ca\") pod \"f0bc9c01-1d10-4e3d-9876-f24637b968ef\" (UID: \"f0bc9c01-1d10-4e3d-9876-f24637b968ef\") "
Oct 30 21:00:21.790917 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:21.788697    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0bc9c01-1d10-4e3d-9876-f24637b968ef-maas-api-service-ca" pod "f0bc9c01-1d10-4e3d-9876-f24637b968ef" (UID: "f0bc9c01-1d10-4e3d-9876-f24637b968ef"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:00:21.794657 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:21.793695    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0bc9c01-1d10-4e3d-9876-f24637b968ef-kube-api-access-7mjmq" pod "f0bc9c01-1d10-4e3d-9876-f24637b968ef" (UID: "f0bc9c01-1d10-4e3d-9876-f24637b968ef"). InnerVolumeSpecName "kube-api-access-7mjmq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:00:21.800923 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:21.800898    4521 scope.go:122] "RemoveContainer" containerID="27851c66fa4928bf0ec43444e747900dba7f4d758608875183cc68c7541d8ecf"
Oct 30 21:00:21.878467 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:21.876166    4521 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-tqd8s"
Oct 30 21:00:21.878467 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:21.877258    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-tqd8s"
Oct 30 21:00:21.878467 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:21.878115    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-tqd8s"
Oct 30 21:00:21.883890 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:21.883860    4521 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0bc9c01-1d10-4e3d-9876-f24637b968ef-maas-api-service-ca\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:00:21.883890 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:21.883888    4521 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mjmq\" (UniqueName: \"kubernetes.io/projected/f0bc9c01-1d10-4e3d-9876-f24637b968ef-kube-api-access-7mjmq\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:00:22.096914 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:22.096877    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c4b379-29889900-sbvxl"
Oct 30 21:00:22.123602 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:22.123496    4521 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-6ggs9"
Oct 30 21:00:22.125542 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:22.125403    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-6ggs9"
Oct 30 21:00:22.132754 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:22.132512    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-6ggs9"
Oct 30 21:00:22.134245 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:22.134214    4521 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-65968b-29889900-hkfkj"]
Oct 30 21:00:22.138735 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:22.138713    4521 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-65968b-29889900-hkfkj"]
Oct 30 21:00:22.228945 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:22.226170    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e319e8b-6373-4ad3-b549-086f4a0c0f25-kube-api-access-7w2zc\" (UniqueName: \"kubernetes.io/projected/1e319e8b-6373-4ad3-b549-086f4a0c0f25-kube-api-access-7w2zc\") pod \"1e319e8b-6373-4ad3-b549-086f4a0c0f25\" (UID: \"1e319e8b-6373-4ad3-b549-086f4a0c0f25\") "
Oct 30 21:00:22.228945 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:22.226363    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e319e8b-6373-4ad3-b549-086f4a0c0f25-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e319e8b-6373-4ad3-b549-086f4a0c0f25-maas-api-service-ca\") pod \"1e319e8b-6373-4ad3-b549-086f4a0c0f25\" (UID: \"1e319e8b-6373-4ad3-b549-086f4a0c0f25\") "
Oct 30 21:00:22.228945 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:22.226945    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e319e8b-6373-4ad3-b549-086f4a0c0f25-maas-api-service-ca" pod "1e319e8b-6373-4ad3-b549-086f4a0c0f25" (UID: "1e319e8b-6373-4ad3-b549-086f4a0c0f25"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:00:22.238502 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:22.238472    4521 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e319e8b-6373-4ad3-b549-086f4a0c0f25-maas-api-service-ca\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:00:22.243939 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:22.243892    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e319e8b-6373-4ad3-b549-086f4a0c0f25-kube-api-access-7w2zc" pod "1e319e8b-6373-4ad3-b549-086f4a0c0f25" (UID: "1e319e8b-6373-4ad3-b549-086f4a0c0f25"). InnerVolumeSpecName "kube-api-access-7w2zc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:00:22.340480 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:22.340439    4521 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7w2zc\" (UniqueName: \"kubernetes.io/projected/1e319e8b-6373-4ad3-b549-086f4a0c0f25-kube-api-access-7w2zc\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:00:22.520952 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:22.520900    4521 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-lp6sk"
Oct 30 21:00:22.521848 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:22.521790    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-lp6sk"
Oct 30 21:00:22.524458 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:22.524392    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-lp6sk"
Oct 30 21:00:22.621087 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:22.621052    4521 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-5xzm6"
Oct 30 21:00:22.621512 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:22.621485    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-5xzm6"
Oct 30 21:00:22.631478 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:22.631445    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-5xzm6"
Oct 30 21:00:22.794819 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:22.794737    4521 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49b3ef08-eca0-4a62-885d-bcf8fc23cb6d" path="/var/lib/kubelet/pods/49b3ef08-eca0-4a62-885d-bcf8fc23cb6d/volumes"
Oct 30 21:00:22.795268 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:22.795247    4521 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0bc9c01-1d10-4e3d-9876-f24637b968ef" path="/var/lib/kubelet/pods/f0bc9c01-1d10-4e3d-9876-f24637b968ef/volumes"
Oct 30 21:00:22.814310 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:22.814272    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c4b379-29889900-sbvxl" event={"ID":"1e319e8b-6373-4ad3-b549-086f4a0c0f25","Type":"ContainerDied","Data":"2f326416916de7c833ef6b0775486e7e4c9b6ad4e47ca66d3a05eee890e13a1b"}
Oct 30 21:00:22.814310 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:22.814351    4521 scope.go:122] "RemoveContainer" containerID="2bb6974367a84213ca55ee650635b48cfed923074ab790f99c266c7e11892084"
Oct 30 21:00:22.818535 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:22.814917    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c4b379-29889900-sbvxl"
Oct 30 21:00:22.842409 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:22.842372    4521 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c4b379-29889900-sbvxl"]
Oct 30 21:00:22.850751 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:22.850714    4521 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c4b379-29889900-sbvxl"]
Oct 30 21:00:24.132286 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:24.132251    4521 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-5a8f94/distinct-w3-5a8f94-kserve-5f8b544d59-mf95j"
Oct 30 21:00:24.148779 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:24.148746    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5a8f94/distinct-w3-5a8f94-kserve-5f8b544d59-mf95j"
Oct 30 21:00:24.793339 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:24.793296    4521 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e319e8b-6373-4ad3-b549-086f4a0c0f25" path="/var/lib/kubelet/pods/1e319e8b-6373-4ad3-b549-086f4a0c0f25/volumes"
Oct 30 21:00:25.392107 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:25.392067    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4kxb"]
Oct 30 21:00:25.780866 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:25.780828    4521 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-f4lmh"]
Oct 30 21:00:25.781409 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:25.781389    4521 cpu_manager.go:414] "containerMap: removing container" podUID="4149346b-deed-426d-9b80-e47822d8b731" containerName="revoke-keys"
Oct 30 21:00:25.781409 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:25.781409    4521 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4149346b-deed-426d-9b80-e47822d8b731" containerName="revoke-keys"
Oct 30 21:00:25.781585 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:25.781423    4521 cpu_manager.go:414] "containerMap: removing container" podUID="49b3ef08-eca0-4a62-885d-bcf8fc23cb6d" containerName="cleanup"
Oct 30 21:00:25.781585 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:25.781431    4521 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49b3ef08-eca0-4a62-885d-bcf8fc23cb6d" containerName="cleanup"
Oct 30 21:00:25.781585 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:25.781443    4521 cpu_manager.go:414] "containerMap: removing container" podUID="f0bc9c01-1d10-4e3d-9876-f24637b968ef" containerName="cleanup"
Oct 30 21:00:25.781585 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:25.781450    4521 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0bc9c01-1d10-4e3d-9876-f24637b968ef" containerName="cleanup"
Oct 30 21:00:25.781585 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:25.781464    4521 cpu_manager.go:414] "containerMap: removing container" podUID="f0bc9c01-1d10-4e3d-9876-f24637b968ef" containerName="cleanup"
Oct 30 21:00:25.781585 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:25.781473    4521 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0bc9c01-1d10-4e3d-9876-f24637b968ef" containerName="cleanup"
Oct 30 21:00:25.781585 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:25.781503    4521 cpu_manager.go:414] "containerMap: removing container" podUID="1e319e8b-6373-4ad3-b549-086f4a0c0f25" containerName="cleanup"
Oct 30 21:00:25.781585 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:25.781509    4521 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e319e8b-6373-4ad3-b549-086f4a0c0f25" containerName="cleanup"
Oct 30 21:00:25.781585 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:25.781526    4521 cpu_manager.go:414] "containerMap: removing container" podUID="49b3ef08-eca0-4a62-885d-bcf8fc23cb6d" containerName="cleanup"
Oct 30 21:00:25.781585 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:25.781533    4521 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49b3ef08-eca0-4a62-885d-bcf8fc23cb6d" containerName="cleanup"
Oct 30 21:00:25.781585 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:25.781551    4521 cpu_manager.go:414] "containerMap: removing container" podUID="1e319e8b-6373-4ad3-b549-086f4a0c0f25" containerName="cleanup"
Oct 30 21:00:25.781585 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:25.781559    4521 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e319e8b-6373-4ad3-b549-086f4a0c0f25" containerName="cleanup"
Oct 30 21:00:25.781585 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:25.781570    4521 cpu_manager.go:414] "containerMap: removing container" podUID="1e319e8b-6373-4ad3-b549-086f4a0c0f25" containerName="cleanup"
Oct 30 21:00:25.781585 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:25.781577    4521 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e319e8b-6373-4ad3-b549-086f4a0c0f25" containerName="cleanup"
Oct 30 21:00:25.782225 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:25.781713    4521 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e319e8b-6373-4ad3-b549-086f4a0c0f25" containerName="cleanup"
Oct 30 21:00:25.782225 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:25.781726    4521 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e319e8b-6373-4ad3-b549-086f4a0c0f25" containerName="cleanup"
Oct 30 21:00:25.782225 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:25.781737    4521 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0bc9c01-1d10-4e3d-9876-f24637b968ef" containerName="cleanup"
Oct 30 21:00:25.782225 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:25.781747    4521 memory_manager.go:365] "RemoveStaleState removing state" podUID="4149346b-deed-426d-9b80-e47822d8b731" containerName="revoke-keys"
Oct 30 21:00:25.782225 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:25.781757    4521 memory_manager.go:365] "RemoveStaleState removing state" podUID="49b3ef08-eca0-4a62-885d-bcf8fc23cb6d" containerName="cleanup"
Oct 30 21:00:25.782225 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:25.781765    4521 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0bc9c01-1d10-4e3d-9876-f24637b968ef" containerName="cleanup"
Oct 30 21:00:25.782225 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:25.781779    4521 memory_manager.go:365] "RemoveStaleState removing state" podUID="49b3ef08-eca0-4a62-885d-bcf8fc23cb6d" containerName="cleanup"
Oct 30 21:00:25.787143 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:25.787119    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-f4lmh"
Oct 30 21:00:25.798056 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:25.798028    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-f4lmh"]
Oct 30 21:00:25.887652 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:25.887613    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9cb1b1d5-a595-403d-9a53-0560efac6ab7-istio-podinfo\") pod \"e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-f4lmh\" (UID: \"9cb1b1d5-a595-403d-9a53-0560efac6ab7\") " pod="openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-f4lmh"
Oct 30 21:00:25.887861 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:25.887682    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cb1b1d5-a595-403d-9a53-0560efac6ab7-credential-socket\") pod \"e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-f4lmh\" (UID: \"9cb1b1d5-a595-403d-9a53-0560efac6ab7\") " pod="openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-f4lmh"
Oct 30 21:00:25.887861 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:25.887751    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9cb1b1d5-a595-403d-9a53-0560efac6ab7-istio-envoy\") pod \"e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-f4lmh\" (UID: \"9cb1b1d5-a595-403d-9a53-0560efac6ab7\") " pod="openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-f4lmh"
Oct 30 21:00:25.887984 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:25.887872    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cb1b1d5-a595-403d-9a53-0560efac6ab7-workload-socket\") pod \"e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-f4lmh\" (UID: \"9cb1b1d5-a595-403d-9a53-0560efac6ab7\") " pod="openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-f4lmh"
Oct 30 21:00:25.888123 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:25.888012    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cb1b1d5-a595-403d-9a53-0560efac6ab7-workload-certs\") pod \"e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-f4lmh\" (UID: \"9cb1b1d5-a595-403d-9a53-0560efac6ab7\") " pod="openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-f4lmh"
Oct 30 21:00:25.888123 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:25.888036    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9cb1b1d5-a595-403d-9a53-0560efac6ab7-istio-data\") pod \"e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-f4lmh\" (UID: \"9cb1b1d5-a595-403d-9a53-0560efac6ab7\") " pod="openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-f4lmh"
Oct 30 21:00:25.888123 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:25.888070    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9cb1b1d5-a595-403d-9a53-0560efac6ab7-istiod-ca-cert\") pod \"e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-f4lmh\" (UID: \"9cb1b1d5-a595-403d-9a53-0560efac6ab7\") " pod="openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-f4lmh"
Oct 30 21:00:25.888274 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:25.888159    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9cb1b1d5-a595-403d-9a53-0560efac6ab7-istio-token\") pod \"e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-f4lmh\" (UID: \"9cb1b1d5-a595-403d-9a53-0560efac6ab7\") " pod="openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-f4lmh"
Oct 30 21:00:25.888274 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:25.888202    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6blzh\" (UniqueName: \"kubernetes.io/projected/9cb1b1d5-a595-403d-9a53-0560efac6ab7-kube-api-access-6blzh\") pod \"e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-f4lmh\" (UID: \"9cb1b1d5-a595-403d-9a53-0560efac6ab7\") " pod="openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-f4lmh"
Oct 30 21:00:25.989925 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:25.989887    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cb1b1d5-a595-403d-9a53-0560efac6ab7-workload-certs\") pod \"e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-f4lmh\" (UID: \"9cb1b1d5-a595-403d-9a53-0560efac6ab7\") " pod="openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-f4lmh"
Oct 30 21:00:25.990093 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:25.989932    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9cb1b1d5-a595-403d-9a53-0560efac6ab7-istio-data\") pod \"e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-f4lmh\" (UID: \"9cb1b1d5-a595-403d-9a53-0560efac6ab7\") " pod="openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-f4lmh"
Oct 30 21:00:25.990093 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:25.989956    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9cb1b1d5-a595-403d-9a53-0560efac6ab7-istiod-ca-cert\") pod \"e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-f4lmh\" (UID: \"9cb1b1d5-a595-403d-9a53-0560efac6ab7\") " pod="openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-f4lmh"
Oct 30 21:00:25.990204 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:25.990148    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9cb1b1d5-a595-403d-9a53-0560efac6ab7-istio-token\") pod \"e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-f4lmh\" (UID: \"9cb1b1d5-a595-403d-9a53-0560efac6ab7\") " pod="openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-f4lmh"
Oct 30 21:00:25.990204 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:25.990181    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6blzh\" (UniqueName: \"kubernetes.io/projected/9cb1b1d5-a595-403d-9a53-0560efac6ab7-kube-api-access-6blzh\") pod \"e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-f4lmh\" (UID: \"9cb1b1d5-a595-403d-9a53-0560efac6ab7\") " pod="openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-f4lmh"
Oct 30 21:00:25.990313 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:25.990256    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9cb1b1d5-a595-403d-9a53-0560efac6ab7-istio-podinfo\") pod \"e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-f4lmh\" (UID: \"9cb1b1d5-a595-403d-9a53-0560efac6ab7\") " pod="openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-f4lmh"
Oct 30 21:00:25.990407 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:25.990359    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cb1b1d5-a595-403d-9a53-0560efac6ab7-credential-socket\") pod \"e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-f4lmh\" (UID: \"9cb1b1d5-a595-403d-9a53-0560efac6ab7\") " pod="openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-f4lmh"
Oct 30 21:00:25.990407 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:25.990386    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9cb1b1d5-a595-403d-9a53-0560efac6ab7-istio-envoy\") pod \"e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-f4lmh\" (UID: \"9cb1b1d5-a595-403d-9a53-0560efac6ab7\") " pod="openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-f4lmh"
Oct 30 21:00:25.990571 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:25.990453    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cb1b1d5-a595-403d-9a53-0560efac6ab7-workload-socket\") pod \"e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-f4lmh\" (UID: \"9cb1b1d5-a595-403d-9a53-0560efac6ab7\") " pod="openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-f4lmh"
Oct 30 21:00:25.990633 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:25.990565    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cb1b1d5-a595-403d-9a53-0560efac6ab7-workload-certs\") pod \"e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-f4lmh\" (UID: \"9cb1b1d5-a595-403d-9a53-0560efac6ab7\") " pod="openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-f4lmh"
Oct 30 21:00:25.990633 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:25.990611    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cb1b1d5-a595-403d-9a53-0560efac6ab7-credential-socket\") pod \"e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-f4lmh\" (UID: \"9cb1b1d5-a595-403d-9a53-0560efac6ab7\") " pod="openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-f4lmh"
Oct 30 21:00:25.990739 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:25.990388    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9cb1b1d5-a595-403d-9a53-0560efac6ab7-istio-data\") pod \"e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-f4lmh\" (UID: \"9cb1b1d5-a595-403d-9a53-0560efac6ab7\") " pod="openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-f4lmh"
Oct 30 21:00:25.990832 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:25.990790    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cb1b1d5-a595-403d-9a53-0560efac6ab7-workload-socket\") pod \"e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-f4lmh\" (UID: \"9cb1b1d5-a595-403d-9a53-0560efac6ab7\") " pod="openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-f4lmh"
Oct 30 21:00:25.991268 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:25.991245    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9cb1b1d5-a595-403d-9a53-0560efac6ab7-istiod-ca-cert\") pod \"e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-f4lmh\" (UID: \"9cb1b1d5-a595-403d-9a53-0560efac6ab7\") " pod="openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-f4lmh"
Oct 30 21:00:25.992775 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:25.992750    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9cb1b1d5-a595-403d-9a53-0560efac6ab7-istio-envoy\") pod \"e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-f4lmh\" (UID: \"9cb1b1d5-a595-403d-9a53-0560efac6ab7\") " pod="openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-f4lmh"
Oct 30 21:00:25.993044 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:25.993023    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9cb1b1d5-a595-403d-9a53-0560efac6ab7-istio-podinfo\") pod \"e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-f4lmh\" (UID: \"9cb1b1d5-a595-403d-9a53-0560efac6ab7\") " pod="openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-f4lmh"
Oct 30 21:00:26.005983 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:26.005831    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9cb1b1d5-a595-403d-9a53-0560efac6ab7-istio-token\") pod \"e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-f4lmh\" (UID: \"9cb1b1d5-a595-403d-9a53-0560efac6ab7\") " pod="openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-f4lmh"
Oct 30 21:00:26.006254 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:26.006230    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6blzh\" (UniqueName: \"kubernetes.io/projected/9cb1b1d5-a595-403d-9a53-0560efac6ab7-kube-api-access-6blzh\") pod \"e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-f4lmh\" (UID: \"9cb1b1d5-a595-403d-9a53-0560efac6ab7\") " pod="openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-f4lmh"
Oct 30 21:00:26.104429 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:26.104047    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-f4lmh"
Oct 30 21:00:26.477422 ip-10-0-131-125 kubenswrapper[4521]: W1030 21:00:26.477379    4521 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cb1b1d5_a595_403d_9a53_0560efac6ab7.slice/crio-2692070e31abaf16b6b68ccdfbbd671354a0d45df2119badad4bc0ba3ae0fab5 WatchSource:0}: Error finding container 2692070e31abaf16b6b68ccdfbbd671354a0d45df2119badad4bc0ba3ae0fab5: Status 404 returned error can't find the container with id 2692070e31abaf16b6b68ccdfbbd671354a0d45df2119badad4bc0ba3ae0fab5
Oct 30 21:00:26.478740 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:26.478711    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-f4lmh"]
Oct 30 21:00:26.480445 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:26.480415    4521 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:00:26.480561 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:26.480473    4521 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:00:26.480561 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:26.480502    4521 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:00:26.929400 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:26.928686    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-f4lmh" event={"ID":"9cb1b1d5-a595-403d-9a53-0560efac6ab7","Type":"ContainerStarted","Data":"b0bcdf545b4a406ab49a1ee5cb8638b9b0323af2615ee83b2080899fc3bddc98"}
Oct 30 21:00:26.929400 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:26.928753    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-f4lmh" event={"ID":"9cb1b1d5-a595-403d-9a53-0560efac6ab7","Type":"ContainerStarted","Data":"2692070e31abaf16b6b68ccdfbbd671354a0d45df2119badad4bc0ba3ae0fab5"}
Oct 30 21:00:26.987347 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:26.985218    4521 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-f4lmh" podStartSLOduration=1.985195077 podStartE2EDuration="1.985195077s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:00:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:00:26.970948042 +0000 UTC m=+1294.316569318" watchObservedRunningTime="2026-10-30 21:00:26.985195077 +0000 UTC m=+1294.330816317"
Oct 30 21:00:27.107724 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:27.107689    4521 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-f4lmh"
Oct 30 21:00:27.112708 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:27.112678    4521 patch_prober.go:29] interesting pod/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-f4lmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Oct 30 21:00:27.112957 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:27.112929    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-f4lmh" podUID="9cb1b1d5-a595-403d-9a53-0560efac6ab7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Oct 30 21:00:28.106349 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:28.105083    4521 patch_prober.go:29] interesting pod/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-f4lmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Oct 30 21:00:28.106349 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:28.105148    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-f4lmh" podUID="9cb1b1d5-a595-403d-9a53-0560efac6ab7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Oct 30 21:00:29.105229 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:29.105190    4521 patch_prober.go:29] interesting pod/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-f4lmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Oct 30 21:00:29.105462 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:29.105247    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-f4lmh" podUID="9cb1b1d5-a595-403d-9a53-0560efac6ab7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Oct 30 21:00:30.105029 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:30.104991    4521 patch_prober.go:29] interesting pod/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-f4lmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Oct 30 21:00:30.105470 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:30.105044    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-f4lmh" podUID="9cb1b1d5-a595-403d-9a53-0560efac6ab7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Oct 30 21:00:31.105618 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:31.105578    4521 patch_prober.go:29] interesting pod/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-f4lmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Oct 30 21:00:31.106027 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:31.105651    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-f4lmh" podUID="9cb1b1d5-a595-403d-9a53-0560efac6ab7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Oct 30 21:00:32.105991 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:32.105953    4521 patch_prober.go:29] interesting pod/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-f4lmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Oct 30 21:00:32.106438 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:32.106026    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-f4lmh" podUID="9cb1b1d5-a595-403d-9a53-0560efac6ab7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Oct 30 21:00:33.105194 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:33.105158    4521 patch_prober.go:29] interesting pod/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-f4lmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Oct 30 21:00:33.105400 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:33.105225    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-f4lmh" podUID="9cb1b1d5-a595-403d-9a53-0560efac6ab7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Oct 30 21:00:34.267605 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:34.267558    4521 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-f4lmh"
Oct 30 21:00:34.268798 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:34.268770    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-f4lmh"
Oct 30 21:00:34.269503 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:34.269478    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-f4lmh"
Oct 30 21:00:39.548477 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:39.548433    4521 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-32a969/distinct-2-w2-32a969-kserve-5c7845448b-vt5gw"]
Oct 30 21:00:39.549244 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:39.549212    4521 cpu_manager.go:414] "containerMap: removing container" podUID="49b3ef08-eca0-4a62-885d-bcf8fc23cb6d" containerName="cleanup"
Oct 30 21:00:39.549244 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:39.549238    4521 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49b3ef08-eca0-4a62-885d-bcf8fc23cb6d" containerName="cleanup"
Oct 30 21:00:39.549898 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:39.549263    4521 cpu_manager.go:414] "containerMap: removing container" podUID="f0bc9c01-1d10-4e3d-9876-f24637b968ef" containerName="cleanup"
Oct 30 21:00:39.549898 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:39.549270    4521 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0bc9c01-1d10-4e3d-9876-f24637b968ef" containerName="cleanup"
Oct 30 21:00:39.549898 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:39.549502    4521 memory_manager.go:365] "RemoveStaleState removing state" podUID="49b3ef08-eca0-4a62-885d-bcf8fc23cb6d" containerName="cleanup"
Oct 30 21:00:39.549898 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:39.549521    4521 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0bc9c01-1d10-4e3d-9876-f24637b968ef" containerName="cleanup"
Oct 30 21:00:39.549898 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:39.549541    4521 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e319e8b-6373-4ad3-b549-086f4a0c0f25" containerName="cleanup"
Oct 30 21:00:39.554959 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:39.554937    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-32a969/distinct-2-w2-32a969-kserve-5c7845448b-vt5gw"
Oct 30 21:00:39.563286 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:39.563259    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-32a969/distinct-2-w2-32a969-kserve-5c7845448b-vt5gw"]
Oct 30 21:00:39.564945 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:39.564923    4521 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-32a969\"/\"distinct-2-w2-32a969-kserve-self-signed-certs\""
Oct 30 21:00:39.588340 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:39.588297    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9sbbt\" (UniqueName: \"kubernetes.io/projected/d430ed01-65d1-43a6-a0c4-ab82212376a9-kube-api-access-9sbbt\") pod \"distinct-2-w2-32a969-kserve-5c7845448b-vt5gw\" (UID: \"d430ed01-65d1-43a6-a0c4-ab82212376a9\") " pod="e2e-models-e2e-worker-w2-32a969/distinct-2-w2-32a969-kserve-5c7845448b-vt5gw"
Oct 30 21:00:39.588495 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:39.588391    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d430ed01-65d1-43a6-a0c4-ab82212376a9-dshm\") pod \"distinct-2-w2-32a969-kserve-5c7845448b-vt5gw\" (UID: \"d430ed01-65d1-43a6-a0c4-ab82212376a9\") " pod="e2e-models-e2e-worker-w2-32a969/distinct-2-w2-32a969-kserve-5c7845448b-vt5gw"
Oct 30 21:00:39.588495 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:39.588462    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d430ed01-65d1-43a6-a0c4-ab82212376a9-home\") pod \"distinct-2-w2-32a969-kserve-5c7845448b-vt5gw\" (UID: \"d430ed01-65d1-43a6-a0c4-ab82212376a9\") " pod="e2e-models-e2e-worker-w2-32a969/distinct-2-w2-32a969-kserve-5c7845448b-vt5gw"
Oct 30 21:00:39.588612 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:39.588519    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d430ed01-65d1-43a6-a0c4-ab82212376a9-tls-certs\") pod \"distinct-2-w2-32a969-kserve-5c7845448b-vt5gw\" (UID: \"d430ed01-65d1-43a6-a0c4-ab82212376a9\") " pod="e2e-models-e2e-worker-w2-32a969/distinct-2-w2-32a969-kserve-5c7845448b-vt5gw"
Oct 30 21:00:39.588612 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:39.588556    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d430ed01-65d1-43a6-a0c4-ab82212376a9-model-cache\") pod \"distinct-2-w2-32a969-kserve-5c7845448b-vt5gw\" (UID: \"d430ed01-65d1-43a6-a0c4-ab82212376a9\") " pod="e2e-models-e2e-worker-w2-32a969/distinct-2-w2-32a969-kserve-5c7845448b-vt5gw"
Oct 30 21:00:39.588723 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:39.588611    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d430ed01-65d1-43a6-a0c4-ab82212376a9-tmp-dir\") pod \"distinct-2-w2-32a969-kserve-5c7845448b-vt5gw\" (UID: \"d430ed01-65d1-43a6-a0c4-ab82212376a9\") " pod="e2e-models-e2e-worker-w2-32a969/distinct-2-w2-32a969-kserve-5c7845448b-vt5gw"
Oct 30 21:00:39.691344 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:39.691144    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d430ed01-65d1-43a6-a0c4-ab82212376a9-tmp-dir\") pod \"distinct-2-w2-32a969-kserve-5c7845448b-vt5gw\" (UID: \"d430ed01-65d1-43a6-a0c4-ab82212376a9\") " pod="e2e-models-e2e-worker-w2-32a969/distinct-2-w2-32a969-kserve-5c7845448b-vt5gw"
Oct 30 21:00:39.691344 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:39.691269    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sbbt\" (UniqueName: \"kubernetes.io/projected/d430ed01-65d1-43a6-a0c4-ab82212376a9-kube-api-access-9sbbt\") pod \"distinct-2-w2-32a969-kserve-5c7845448b-vt5gw\" (UID: \"d430ed01-65d1-43a6-a0c4-ab82212376a9\") " pod="e2e-models-e2e-worker-w2-32a969/distinct-2-w2-32a969-kserve-5c7845448b-vt5gw"
Oct 30 21:00:39.691607 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:39.691363    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d430ed01-65d1-43a6-a0c4-ab82212376a9-dshm\") pod \"distinct-2-w2-32a969-kserve-5c7845448b-vt5gw\" (UID: \"d430ed01-65d1-43a6-a0c4-ab82212376a9\") " pod="e2e-models-e2e-worker-w2-32a969/distinct-2-w2-32a969-kserve-5c7845448b-vt5gw"
Oct 30 21:00:39.691607 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:39.691448    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d430ed01-65d1-43a6-a0c4-ab82212376a9-home\") pod \"distinct-2-w2-32a969-kserve-5c7845448b-vt5gw\" (UID: \"d430ed01-65d1-43a6-a0c4-ab82212376a9\") " pod="e2e-models-e2e-worker-w2-32a969/distinct-2-w2-32a969-kserve-5c7845448b-vt5gw"
Oct 30 21:00:39.691607 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:39.691510    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d430ed01-65d1-43a6-a0c4-ab82212376a9-tls-certs\") pod \"distinct-2-w2-32a969-kserve-5c7845448b-vt5gw\" (UID: \"d430ed01-65d1-43a6-a0c4-ab82212376a9\") " pod="e2e-models-e2e-worker-w2-32a969/distinct-2-w2-32a969-kserve-5c7845448b-vt5gw"
Oct 30 21:00:39.691607 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:39.691545    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d430ed01-65d1-43a6-a0c4-ab82212376a9-model-cache\") pod \"distinct-2-w2-32a969-kserve-5c7845448b-vt5gw\" (UID: \"d430ed01-65d1-43a6-a0c4-ab82212376a9\") " pod="e2e-models-e2e-worker-w2-32a969/distinct-2-w2-32a969-kserve-5c7845448b-vt5gw"
Oct 30 21:00:39.692033 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:39.692006    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d430ed01-65d1-43a6-a0c4-ab82212376a9-model-cache\") pod \"distinct-2-w2-32a969-kserve-5c7845448b-vt5gw\" (UID: \"d430ed01-65d1-43a6-a0c4-ab82212376a9\") " pod="e2e-models-e2e-worker-w2-32a969/distinct-2-w2-32a969-kserve-5c7845448b-vt5gw"
Oct 30 21:00:39.694293 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:39.693226    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d430ed01-65d1-43a6-a0c4-ab82212376a9-tmp-dir\") pod \"distinct-2-w2-32a969-kserve-5c7845448b-vt5gw\" (UID: \"d430ed01-65d1-43a6-a0c4-ab82212376a9\") " pod="e2e-models-e2e-worker-w2-32a969/distinct-2-w2-32a969-kserve-5c7845448b-vt5gw"
Oct 30 21:00:39.694511 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:39.693516    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d430ed01-65d1-43a6-a0c4-ab82212376a9-home\") pod \"distinct-2-w2-32a969-kserve-5c7845448b-vt5gw\" (UID: \"d430ed01-65d1-43a6-a0c4-ab82212376a9\") " pod="e2e-models-e2e-worker-w2-32a969/distinct-2-w2-32a969-kserve-5c7845448b-vt5gw"
Oct 30 21:00:39.700237 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:39.700214    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d430ed01-65d1-43a6-a0c4-ab82212376a9-dshm\") pod \"distinct-2-w2-32a969-kserve-5c7845448b-vt5gw\" (UID: \"d430ed01-65d1-43a6-a0c4-ab82212376a9\") " pod="e2e-models-e2e-worker-w2-32a969/distinct-2-w2-32a969-kserve-5c7845448b-vt5gw"
Oct 30 21:00:39.714096 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:39.714062    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d430ed01-65d1-43a6-a0c4-ab82212376a9-tls-certs\") pod \"distinct-2-w2-32a969-kserve-5c7845448b-vt5gw\" (UID: \"d430ed01-65d1-43a6-a0c4-ab82212376a9\") " pod="e2e-models-e2e-worker-w2-32a969/distinct-2-w2-32a969-kserve-5c7845448b-vt5gw"
Oct 30 21:00:39.723824 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:39.723787    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9sbbt\" (UniqueName: \"kubernetes.io/projected/d430ed01-65d1-43a6-a0c4-ab82212376a9-kube-api-access-9sbbt\") pod \"distinct-2-w2-32a969-kserve-5c7845448b-vt5gw\" (UID: \"d430ed01-65d1-43a6-a0c4-ab82212376a9\") " pod="e2e-models-e2e-worker-w2-32a969/distinct-2-w2-32a969-kserve-5c7845448b-vt5gw"
Oct 30 21:00:39.883418 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:39.883310    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-32a969/distinct-2-w2-32a969-kserve-5c7845448b-vt5gw"
Oct 30 21:00:40.185041 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:40.185006    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-32a969/distinct-2-w2-32a969-kserve-5c7845448b-vt5gw"]
Oct 30 21:00:40.185795 ip-10-0-131-125 kubenswrapper[4521]: W1030 21:00:40.185770    4521 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd430ed01_65d1_43a6_a0c4_ab82212376a9.slice/crio-57330fffeb199ff8e3bca7ddbe80a07100580731ba35b750a638aaa8022b13f5 WatchSource:0}: Error finding container 57330fffeb199ff8e3bca7ddbe80a07100580731ba35b750a638aaa8022b13f5: Status 404 returned error can't find the container with id 57330fffeb199ff8e3bca7ddbe80a07100580731ba35b750a638aaa8022b13f5
Oct 30 21:00:41.012352 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:41.012292    4521 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-39fb96-openshift-default-5f55489475-ctbdp"]
Oct 30 21:00:41.016464 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:41.016440    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-39fb96-openshift-default-5f55489475-ctbdp"
Oct 30 21:00:41.026027 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:41.025754    4521 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-39fb96-openshift-default-dockercfg-t7kpm\""
Oct 30 21:00:41.031130 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:41.031102    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-39fb96-openshift-default-5f55489475-ctbdp"]
Oct 30 21:00:41.108541 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:41.108509    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55990440-6bb2-4ce7-945f-6ada2906a855-istio-envoy\") pod \"e2e-worker-w6-39fb96-openshift-default-5f55489475-ctbdp\" (UID: \"55990440-6bb2-4ce7-945f-6ada2906a855\") " pod="openshift-ingress/e2e-worker-w6-39fb96-openshift-default-5f55489475-ctbdp"
Oct 30 21:00:41.108731 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:41.108551    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55990440-6bb2-4ce7-945f-6ada2906a855-workload-socket\") pod \"e2e-worker-w6-39fb96-openshift-default-5f55489475-ctbdp\" (UID: \"55990440-6bb2-4ce7-945f-6ada2906a855\") " pod="openshift-ingress/e2e-worker-w6-39fb96-openshift-default-5f55489475-ctbdp"
Oct 30 21:00:41.108731 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:41.108634    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctk9j\" (UniqueName: \"kubernetes.io/projected/55990440-6bb2-4ce7-945f-6ada2906a855-kube-api-access-ctk9j\") pod \"e2e-worker-w6-39fb96-openshift-default-5f55489475-ctbdp\" (UID: \"55990440-6bb2-4ce7-945f-6ada2906a855\") " pod="openshift-ingress/e2e-worker-w6-39fb96-openshift-default-5f55489475-ctbdp"
Oct 30 21:00:41.108731 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:41.108682    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55990440-6bb2-4ce7-945f-6ada2906a855-istiod-ca-cert\") pod \"e2e-worker-w6-39fb96-openshift-default-5f55489475-ctbdp\" (UID: \"55990440-6bb2-4ce7-945f-6ada2906a855\") " pod="openshift-ingress/e2e-worker-w6-39fb96-openshift-default-5f55489475-ctbdp"
Oct 30 21:00:41.108887 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:41.108835    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55990440-6bb2-4ce7-945f-6ada2906a855-istio-token\") pod \"e2e-worker-w6-39fb96-openshift-default-5f55489475-ctbdp\" (UID: \"55990440-6bb2-4ce7-945f-6ada2906a855\") " pod="openshift-ingress/e2e-worker-w6-39fb96-openshift-default-5f55489475-ctbdp"
Oct 30 21:00:41.108945 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:41.108903    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55990440-6bb2-4ce7-945f-6ada2906a855-workload-certs\") pod \"e2e-worker-w6-39fb96-openshift-default-5f55489475-ctbdp\" (UID: \"55990440-6bb2-4ce7-945f-6ada2906a855\") " pod="openshift-ingress/e2e-worker-w6-39fb96-openshift-default-5f55489475-ctbdp"
Oct 30 21:00:41.108945 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:41.108939    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55990440-6bb2-4ce7-945f-6ada2906a855-credential-socket\") pod \"e2e-worker-w6-39fb96-openshift-default-5f55489475-ctbdp\" (UID: \"55990440-6bb2-4ce7-945f-6ada2906a855\") " pod="openshift-ingress/e2e-worker-w6-39fb96-openshift-default-5f55489475-ctbdp"
Oct 30 21:00:41.109038 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:41.108961    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55990440-6bb2-4ce7-945f-6ada2906a855-istio-data\") pod \"e2e-worker-w6-39fb96-openshift-default-5f55489475-ctbdp\" (UID: \"55990440-6bb2-4ce7-945f-6ada2906a855\") " pod="openshift-ingress/e2e-worker-w6-39fb96-openshift-default-5f55489475-ctbdp"
Oct 30 21:00:41.109070 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:41.109031    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55990440-6bb2-4ce7-945f-6ada2906a855-istio-podinfo\") pod \"e2e-worker-w6-39fb96-openshift-default-5f55489475-ctbdp\" (UID: \"55990440-6bb2-4ce7-945f-6ada2906a855\") " pod="openshift-ingress/e2e-worker-w6-39fb96-openshift-default-5f55489475-ctbdp"
Oct 30 21:00:41.179619 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:41.179581    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-32a969/distinct-2-w2-32a969-kserve-5c7845448b-vt5gw" event={"ID":"d430ed01-65d1-43a6-a0c4-ab82212376a9","Type":"ContainerStarted","Data":"66aa776bf0145a82d05e5842d09817cbf3e9d4e1b43e9253569b980f73af2e3a"}
Oct 30 21:00:41.179803 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:41.179631    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-32a969/distinct-2-w2-32a969-kserve-5c7845448b-vt5gw" event={"ID":"d430ed01-65d1-43a6-a0c4-ab82212376a9","Type":"ContainerStarted","Data":"57330fffeb199ff8e3bca7ddbe80a07100580731ba35b750a638aaa8022b13f5"}
Oct 30 21:00:41.200184 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:41.200134    4521 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-32a969/distinct-2-w2-32a969-kserve-5c7845448b-vt5gw" podStartSLOduration=2.2001196419999998 podStartE2EDuration="2.200119642s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:00:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:00:41.19911956 +0000 UTC m=+1308.544740814" watchObservedRunningTime="2026-10-30 21:00:41.200119642 +0000 UTC m=+1308.545740930"
Oct 30 21:00:41.209677 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:41.209648    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55990440-6bb2-4ce7-945f-6ada2906a855-workload-certs\") pod \"e2e-worker-w6-39fb96-openshift-default-5f55489475-ctbdp\" (UID: \"55990440-6bb2-4ce7-945f-6ada2906a855\") " pod="openshift-ingress/e2e-worker-w6-39fb96-openshift-default-5f55489475-ctbdp"
Oct 30 21:00:41.209851 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:41.209708    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55990440-6bb2-4ce7-945f-6ada2906a855-credential-socket\") pod \"e2e-worker-w6-39fb96-openshift-default-5f55489475-ctbdp\" (UID: \"55990440-6bb2-4ce7-945f-6ada2906a855\") " pod="openshift-ingress/e2e-worker-w6-39fb96-openshift-default-5f55489475-ctbdp"
Oct 30 21:00:41.209851 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:41.209740    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55990440-6bb2-4ce7-945f-6ada2906a855-istio-data\") pod \"e2e-worker-w6-39fb96-openshift-default-5f55489475-ctbdp\" (UID: \"55990440-6bb2-4ce7-945f-6ada2906a855\") " pod="openshift-ingress/e2e-worker-w6-39fb96-openshift-default-5f55489475-ctbdp"
Oct 30 21:00:41.209851 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:41.209771    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55990440-6bb2-4ce7-945f-6ada2906a855-istio-podinfo\") pod \"e2e-worker-w6-39fb96-openshift-default-5f55489475-ctbdp\" (UID: \"55990440-6bb2-4ce7-945f-6ada2906a855\") " pod="openshift-ingress/e2e-worker-w6-39fb96-openshift-default-5f55489475-ctbdp"
Oct 30 21:00:41.210019 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:41.209857    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55990440-6bb2-4ce7-945f-6ada2906a855-istio-envoy\") pod \"e2e-worker-w6-39fb96-openshift-default-5f55489475-ctbdp\" (UID: \"55990440-6bb2-4ce7-945f-6ada2906a855\") " pod="openshift-ingress/e2e-worker-w6-39fb96-openshift-default-5f55489475-ctbdp"
Oct 30 21:00:41.210019 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:41.209886    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55990440-6bb2-4ce7-945f-6ada2906a855-workload-socket\") pod \"e2e-worker-w6-39fb96-openshift-default-5f55489475-ctbdp\" (UID: \"55990440-6bb2-4ce7-945f-6ada2906a855\") " pod="openshift-ingress/e2e-worker-w6-39fb96-openshift-default-5f55489475-ctbdp"
Oct 30 21:00:41.210019 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:41.209922    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctk9j\" (UniqueName: \"kubernetes.io/projected/55990440-6bb2-4ce7-945f-6ada2906a855-kube-api-access-ctk9j\") pod \"e2e-worker-w6-39fb96-openshift-default-5f55489475-ctbdp\" (UID: \"55990440-6bb2-4ce7-945f-6ada2906a855\") " pod="openshift-ingress/e2e-worker-w6-39fb96-openshift-default-5f55489475-ctbdp"
Oct 30 21:00:41.210019 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:41.209982    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55990440-6bb2-4ce7-945f-6ada2906a855-istiod-ca-cert\") pod \"e2e-worker-w6-39fb96-openshift-default-5f55489475-ctbdp\" (UID: \"55990440-6bb2-4ce7-945f-6ada2906a855\") " pod="openshift-ingress/e2e-worker-w6-39fb96-openshift-default-5f55489475-ctbdp"
Oct 30 21:00:41.210231 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:41.210075    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55990440-6bb2-4ce7-945f-6ada2906a855-istio-token\") pod \"e2e-worker-w6-39fb96-openshift-default-5f55489475-ctbdp\" (UID: \"55990440-6bb2-4ce7-945f-6ada2906a855\") " pod="openshift-ingress/e2e-worker-w6-39fb96-openshift-default-5f55489475-ctbdp"
Oct 30 21:00:41.210287 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:41.210255    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55990440-6bb2-4ce7-945f-6ada2906a855-credential-socket\") pod \"e2e-worker-w6-39fb96-openshift-default-5f55489475-ctbdp\" (UID: \"55990440-6bb2-4ce7-945f-6ada2906a855\") " pod="openshift-ingress/e2e-worker-w6-39fb96-openshift-default-5f55489475-ctbdp"
Oct 30 21:00:41.210544 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:41.210521    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55990440-6bb2-4ce7-945f-6ada2906a855-workload-socket\") pod \"e2e-worker-w6-39fb96-openshift-default-5f55489475-ctbdp\" (UID: \"55990440-6bb2-4ce7-945f-6ada2906a855\") " pod="openshift-ingress/e2e-worker-w6-39fb96-openshift-default-5f55489475-ctbdp"
Oct 30 21:00:41.210846 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:41.210826    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55990440-6bb2-4ce7-945f-6ada2906a855-workload-certs\") pod \"e2e-worker-w6-39fb96-openshift-default-5f55489475-ctbdp\" (UID: \"55990440-6bb2-4ce7-945f-6ada2906a855\") " pod="openshift-ingress/e2e-worker-w6-39fb96-openshift-default-5f55489475-ctbdp"
Oct 30 21:00:41.211087 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:41.211061    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55990440-6bb2-4ce7-945f-6ada2906a855-istio-data\") pod \"e2e-worker-w6-39fb96-openshift-default-5f55489475-ctbdp\" (UID: \"55990440-6bb2-4ce7-945f-6ada2906a855\") " pod="openshift-ingress/e2e-worker-w6-39fb96-openshift-default-5f55489475-ctbdp"
Oct 30 21:00:41.211237 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:41.211210    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55990440-6bb2-4ce7-945f-6ada2906a855-istiod-ca-cert\") pod \"e2e-worker-w6-39fb96-openshift-default-5f55489475-ctbdp\" (UID: \"55990440-6bb2-4ce7-945f-6ada2906a855\") " pod="openshift-ingress/e2e-worker-w6-39fb96-openshift-default-5f55489475-ctbdp"
Oct 30 21:00:41.218363 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:41.212977    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55990440-6bb2-4ce7-945f-6ada2906a855-istio-envoy\") pod \"e2e-worker-w6-39fb96-openshift-default-5f55489475-ctbdp\" (UID: \"55990440-6bb2-4ce7-945f-6ada2906a855\") " pod="openshift-ingress/e2e-worker-w6-39fb96-openshift-default-5f55489475-ctbdp"
Oct 30 21:00:41.218363 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:41.217875    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55990440-6bb2-4ce7-945f-6ada2906a855-istio-podinfo\") pod \"e2e-worker-w6-39fb96-openshift-default-5f55489475-ctbdp\" (UID: \"55990440-6bb2-4ce7-945f-6ada2906a855\") " pod="openshift-ingress/e2e-worker-w6-39fb96-openshift-default-5f55489475-ctbdp"
Oct 30 21:00:41.223148 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:41.223124    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctk9j\" (UniqueName: \"kubernetes.io/projected/55990440-6bb2-4ce7-945f-6ada2906a855-kube-api-access-ctk9j\") pod \"e2e-worker-w6-39fb96-openshift-default-5f55489475-ctbdp\" (UID: \"55990440-6bb2-4ce7-945f-6ada2906a855\") " pod="openshift-ingress/e2e-worker-w6-39fb96-openshift-default-5f55489475-ctbdp"
Oct 30 21:00:41.223148 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:41.223138    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55990440-6bb2-4ce7-945f-6ada2906a855-istio-token\") pod \"e2e-worker-w6-39fb96-openshift-default-5f55489475-ctbdp\" (UID: \"55990440-6bb2-4ce7-945f-6ada2906a855\") " pod="openshift-ingress/e2e-worker-w6-39fb96-openshift-default-5f55489475-ctbdp"
Oct 30 21:00:41.333738 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:41.333636    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-39fb96-openshift-default-5f55489475-ctbdp"
Oct 30 21:00:41.638151 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:41.638123    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-39fb96-openshift-default-5f55489475-ctbdp"]
Oct 30 21:00:41.638460 ip-10-0-131-125 kubenswrapper[4521]: W1030 21:00:41.638429    4521 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55990440_6bb2_4ce7_945f_6ada2906a855.slice/crio-4b3c8f3fca2a92a37a937714474ca054fde5bcb6876b18e37273c7c7eee57264 WatchSource:0}: Error finding container 4b3c8f3fca2a92a37a937714474ca054fde5bcb6876b18e37273c7c7eee57264: Status 404 returned error can't find the container with id 4b3c8f3fca2a92a37a937714474ca054fde5bcb6876b18e37273c7c7eee57264
Oct 30 21:00:41.641655 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:41.641620    4521 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:00:41.641763 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:41.641688    4521 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:00:41.641763 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:41.641715    4521 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:00:41.884164 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:41.884130    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-32a969/distinct-2-w2-32a969-kserve-5c7845448b-vt5gw"
Oct 30 21:00:42.187754 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:42.187719    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-39fb96-openshift-default-5f55489475-ctbdp" event={"ID":"55990440-6bb2-4ce7-945f-6ada2906a855","Type":"ContainerStarted","Data":"f3f770366013af52bebfc6b7500a37d53693245530cfa7a6c52c4aa6ebbbbee0"}
Oct 30 21:00:42.188210 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:42.187776    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-39fb96-openshift-default-5f55489475-ctbdp" event={"ID":"55990440-6bb2-4ce7-945f-6ada2906a855","Type":"ContainerStarted","Data":"4b3c8f3fca2a92a37a937714474ca054fde5bcb6876b18e37273c7c7eee57264"}
Oct 30 21:00:42.211019 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:42.210958    4521 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-39fb96-openshift-default-5f55489475-ctbdp" podStartSLOduration=2.210938412 podStartE2EDuration="2.210938412s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:00:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:00:42.210505242 +0000 UTC m=+1309.556126481" watchObservedRunningTime="2026-10-30 21:00:42.210938412 +0000 UTC m=+1309.556559653"
Oct 30 21:00:42.334139 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:42.334104    4521 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-39fb96-openshift-default-5f55489475-ctbdp"
Oct 30 21:00:42.336070 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:42.336038    4521 patch_prober.go:29] interesting pod/e2e-worker-w6-39fb96-openshift-default-5f55489475-ctbdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 30 21:00:42.336219 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:42.336085    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-39fb96-openshift-default-5f55489475-ctbdp" podUID="55990440-6bb2-4ce7-945f-6ada2906a855" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 30 21:00:43.341042 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:43.340986    4521 patch_prober.go:29] interesting pod/e2e-worker-w6-39fb96-openshift-default-5f55489475-ctbdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 30 21:00:43.341703 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:43.341670    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-39fb96-openshift-default-5f55489475-ctbdp" podUID="55990440-6bb2-4ce7-945f-6ada2906a855" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 30 21:00:44.343388 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:44.342579    4521 patch_prober.go:29] interesting pod/e2e-worker-w6-39fb96-openshift-default-5f55489475-ctbdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 30 21:00:44.343388 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:44.342639    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-39fb96-openshift-default-5f55489475-ctbdp" podUID="55990440-6bb2-4ce7-945f-6ada2906a855" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 30 21:00:45.338348 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:45.337814    4521 patch_prober.go:29] interesting pod/e2e-worker-w6-39fb96-openshift-default-5f55489475-ctbdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 30 21:00:45.338348 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:45.337894    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-39fb96-openshift-default-5f55489475-ctbdp" podUID="55990440-6bb2-4ce7-945f-6ada2906a855" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 30 21:00:46.334786 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:46.334746    4521 patch_prober.go:29] interesting pod/e2e-worker-w6-39fb96-openshift-default-5f55489475-ctbdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 30 21:00:46.335430 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:46.335394    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-39fb96-openshift-default-5f55489475-ctbdp" podUID="55990440-6bb2-4ce7-945f-6ada2906a855" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 30 21:00:46.457723 ip-10-0-131-125 kubenswrapper[4521]: E1030 21:00:46.457602    4521 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lkn2k" containerName="extractor"
Oct 30 21:00:47.338345 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:47.337786    4521 patch_prober.go:29] interesting pod/e2e-worker-w6-39fb96-openshift-default-5f55489475-ctbdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 30 21:00:47.338345 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:47.337870    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-39fb96-openshift-default-5f55489475-ctbdp" podUID="55990440-6bb2-4ce7-945f-6ada2906a855" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 30 21:00:48.151826 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:48.151786    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4kxb"]
Oct 30 21:00:48.334835 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:48.334791    4521 patch_prober.go:29] interesting pod/e2e-worker-w6-39fb96-openshift-default-5f55489475-ctbdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 30 21:00:48.335042 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:48.334877    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-39fb96-openshift-default-5f55489475-ctbdp" podUID="55990440-6bb2-4ce7-945f-6ada2906a855" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 30 21:00:49.956348 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:49.935617    4521 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-32a969/distinct-2-w2-32a969-kserve-5c7845448b-vt5gw"
Oct 30 21:00:50.410410 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:50.409917    4521 patch_prober.go:29] interesting pod/e2e-worker-w6-39fb96-openshift-default-5f55489475-ctbdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:00:50.410410 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:50.410004    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-39fb96-openshift-default-5f55489475-ctbdp" podUID="55990440-6bb2-4ce7-945f-6ada2906a855" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:00:50.438831 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:50.438798    4521 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-32a969/distinct-2-w2-32a969-kserve-5c7845448b-vt5gw"
Oct 30 21:00:50.541344 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:50.539652    4521 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-39fb96-openshift-default-5f55489475-ctbdp"
Oct 30 21:00:50.541344 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:50.541167    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-39fb96-openshift-default-5f55489475-ctbdp"
Oct 30 21:00:50.547590 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:50.547539    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-39fb96-openshift-default-5f55489475-ctbdp"
Oct 30 21:00:50.786346 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:50.777441    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c2d1eb/premium-w4-c2d1eb-kserve-8696fbd9b9-mxqhm" podUID="4bb0f898-b77f-419d-bff4-6413e82ef51d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:00:50.980470 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:50.980422    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-39fb96/premium-w6-39fb96-kserve-b9cf7f75c-96blx" podUID="18ca0a94-c322-41fa-9fd9-0f14419b5075" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:00:51.005445 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:51.005401    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-65968b/premium-w5-65968b-kserve-67977485f7-c9jpv" podUID="92b2dd65-8a83-436d-afa8-50e5f0d11fa0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 30 21:00:51.006128 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:51.005701    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-32a969/simulator-w2-32a969-kserve-6cc8dfc44c-dwhlt" podUID="62a6adf7-d17a-4fec-a707-a2d549b67753" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:00:51.007316 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:51.005737    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a0c5b/premium-w0-4a0c5b-kserve-85dcf89f57-mscdn" podUID="00e42758-77ce-4663-adfa-2202c0aae24a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:00:51.086519 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:51.086408    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a8f94/premium-w3-5a8f94-kserve-bb96b6848-5hk8h" podUID="24ccc37d-027a-4097-8da5-04df417aef94" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:00:51.140452 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:51.140404    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a8f94/distinct-w3-5a8f94-kserve-5f8b544d59-mf95j" podUID="b8ac1cca-4ebd-474c-b32f-a3595a4cd847" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:00:51.394369 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:51.394253    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-32a969/distinct-2-w2-32a969-kserve-5c7845448b-vt5gw"
Oct 30 21:00:53.061359 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:53.061307    4521 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59c65aca-t465g"]
Oct 30 21:00:53.067030 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:53.067007    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59c65aca-t465g"
Oct 30 21:00:53.071069 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:53.071044    4521 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:00:53.071404 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:53.071380    4521 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c45pw\""
Oct 30 21:00:53.078305 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:53.078278    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59c65aca-t465g"]
Oct 30 21:00:53.203668 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:53.203631    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa6b7076-9843-483f-94ce-e61e47536580-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59c65aca-t465g\" (UID: \"fa6b7076-9843-483f-94ce-e61e47536580\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59c65aca-t465g"
Oct 30 21:00:53.203668 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:53.203671    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrkwm\" (UniqueName: \"kubernetes.io/projected/fa6b7076-9843-483f-94ce-e61e47536580-kube-api-access-xrkwm\") pod \"maas-api-revoke-sub-59c65aca-t465g\" (UID: \"fa6b7076-9843-483f-94ce-e61e47536580\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59c65aca-t465g"
Oct 30 21:00:53.203930 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:53.203788    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa6b7076-9843-483f-94ce-e61e47536580-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59c65aca-t465g\" (UID: \"fa6b7076-9843-483f-94ce-e61e47536580\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59c65aca-t465g"
Oct 30 21:00:53.304924 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:53.304829    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa6b7076-9843-483f-94ce-e61e47536580-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59c65aca-t465g\" (UID: \"fa6b7076-9843-483f-94ce-e61e47536580\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59c65aca-t465g"
Oct 30 21:00:53.304924 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:53.304882    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrkwm\" (UniqueName: \"kubernetes.io/projected/fa6b7076-9843-483f-94ce-e61e47536580-kube-api-access-xrkwm\") pod \"maas-api-revoke-sub-59c65aca-t465g\" (UID: \"fa6b7076-9843-483f-94ce-e61e47536580\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59c65aca-t465g"
Oct 30 21:00:53.305129 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:53.304949    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa6b7076-9843-483f-94ce-e61e47536580-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59c65aca-t465g\" (UID: \"fa6b7076-9843-483f-94ce-e61e47536580\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59c65aca-t465g"
Oct 30 21:00:53.305636 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:53.305612    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa6b7076-9843-483f-94ce-e61e47536580-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59c65aca-t465g\" (UID: \"fa6b7076-9843-483f-94ce-e61e47536580\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59c65aca-t465g"
Oct 30 21:00:53.307371 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:53.307349    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa6b7076-9843-483f-94ce-e61e47536580-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59c65aca-t465g\" (UID: \"fa6b7076-9843-483f-94ce-e61e47536580\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59c65aca-t465g"
Oct 30 21:00:53.317859 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:53.317827    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrkwm\" (UniqueName: \"kubernetes.io/projected/fa6b7076-9843-483f-94ce-e61e47536580-kube-api-access-xrkwm\") pod \"maas-api-revoke-sub-59c65aca-t465g\" (UID: \"fa6b7076-9843-483f-94ce-e61e47536580\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59c65aca-t465g"
Oct 30 21:00:53.385962 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:53.385926    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59c65aca-t465g"
Oct 30 21:00:53.454635 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:53.454593    4521 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-5a8f94/distinct-2-w3-5a8f94-kserve-ffcc6449b-hv48k"]
Oct 30 21:00:53.478467 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:53.478435    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-5a8f94/distinct-2-w3-5a8f94-kserve-ffcc6449b-hv48k"
Oct 30 21:00:53.490251 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:53.490228    4521 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-5a8f94\"/\"distinct-2-w3-5a8f94-kserve-self-signed-certs\""
Oct 30 21:00:53.498663 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:53.498627    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-5a8f94/distinct-2-w3-5a8f94-kserve-ffcc6449b-hv48k"]
Oct 30 21:00:53.611026 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:53.610089    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc0f5352-5c44-4c35-adf5-3fffab06b644-tmp-dir\") pod \"distinct-2-w3-5a8f94-kserve-ffcc6449b-hv48k\" (UID: \"bc0f5352-5c44-4c35-adf5-3fffab06b644\") " pod="e2e-models-e2e-worker-w3-5a8f94/distinct-2-w3-5a8f94-kserve-ffcc6449b-hv48k"
Oct 30 21:00:53.611026 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:53.610144    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc0f5352-5c44-4c35-adf5-3fffab06b644-tls-certs\") pod \"distinct-2-w3-5a8f94-kserve-ffcc6449b-hv48k\" (UID: \"bc0f5352-5c44-4c35-adf5-3fffab06b644\") " pod="e2e-models-e2e-worker-w3-5a8f94/distinct-2-w3-5a8f94-kserve-ffcc6449b-hv48k"
Oct 30 21:00:53.611026 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:53.610306    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc0f5352-5c44-4c35-adf5-3fffab06b644-model-cache\") pod \"distinct-2-w3-5a8f94-kserve-ffcc6449b-hv48k\" (UID: \"bc0f5352-5c44-4c35-adf5-3fffab06b644\") " pod="e2e-models-e2e-worker-w3-5a8f94/distinct-2-w3-5a8f94-kserve-ffcc6449b-hv48k"
Oct 30 21:00:53.611026 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:53.610366    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tshfg\" (UniqueName: \"kubernetes.io/projected/bc0f5352-5c44-4c35-adf5-3fffab06b644-kube-api-access-tshfg\") pod \"distinct-2-w3-5a8f94-kserve-ffcc6449b-hv48k\" (UID: \"bc0f5352-5c44-4c35-adf5-3fffab06b644\") " pod="e2e-models-e2e-worker-w3-5a8f94/distinct-2-w3-5a8f94-kserve-ffcc6449b-hv48k"
Oct 30 21:00:53.611026 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:53.610398    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc0f5352-5c44-4c35-adf5-3fffab06b644-home\") pod \"distinct-2-w3-5a8f94-kserve-ffcc6449b-hv48k\" (UID: \"bc0f5352-5c44-4c35-adf5-3fffab06b644\") " pod="e2e-models-e2e-worker-w3-5a8f94/distinct-2-w3-5a8f94-kserve-ffcc6449b-hv48k"
Oct 30 21:00:53.611026 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:53.610417    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc0f5352-5c44-4c35-adf5-3fffab06b644-dshm\") pod \"distinct-2-w3-5a8f94-kserve-ffcc6449b-hv48k\" (UID: \"bc0f5352-5c44-4c35-adf5-3fffab06b644\") " pod="e2e-models-e2e-worker-w3-5a8f94/distinct-2-w3-5a8f94-kserve-ffcc6449b-hv48k"
Oct 30 21:00:53.703639 ip-10-0-131-125 kubenswrapper[4521]: W1030 21:00:53.703602    4521 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa6b7076_9843_483f_94ce_e61e47536580.slice/crio-b84a8d6a304501953525b2b89b29c2eca8b67049b2f9df6c162c0eb887993886 WatchSource:0}: Error finding container b84a8d6a304501953525b2b89b29c2eca8b67049b2f9df6c162c0eb887993886: Status 404 returned error can't find the container with id b84a8d6a304501953525b2b89b29c2eca8b67049b2f9df6c162c0eb887993886
Oct 30 21:00:53.711203 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:53.710851    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59c65aca-t465g"]
Oct 30 21:00:53.719741 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:53.711629    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc0f5352-5c44-4c35-adf5-3fffab06b644-model-cache\") pod \"distinct-2-w3-5a8f94-kserve-ffcc6449b-hv48k\" (UID: \"bc0f5352-5c44-4c35-adf5-3fffab06b644\") " pod="e2e-models-e2e-worker-w3-5a8f94/distinct-2-w3-5a8f94-kserve-ffcc6449b-hv48k"
Oct 30 21:00:53.719741 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:53.711681    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tshfg\" (UniqueName: \"kubernetes.io/projected/bc0f5352-5c44-4c35-adf5-3fffab06b644-kube-api-access-tshfg\") pod \"distinct-2-w3-5a8f94-kserve-ffcc6449b-hv48k\" (UID: \"bc0f5352-5c44-4c35-adf5-3fffab06b644\") " pod="e2e-models-e2e-worker-w3-5a8f94/distinct-2-w3-5a8f94-kserve-ffcc6449b-hv48k"
Oct 30 21:00:53.719741 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:53.711719    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc0f5352-5c44-4c35-adf5-3fffab06b644-home\") pod \"distinct-2-w3-5a8f94-kserve-ffcc6449b-hv48k\" (UID: \"bc0f5352-5c44-4c35-adf5-3fffab06b644\") " pod="e2e-models-e2e-worker-w3-5a8f94/distinct-2-w3-5a8f94-kserve-ffcc6449b-hv48k"
Oct 30 21:00:53.719741 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:53.711755    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc0f5352-5c44-4c35-adf5-3fffab06b644-dshm\") pod \"distinct-2-w3-5a8f94-kserve-ffcc6449b-hv48k\" (UID: \"bc0f5352-5c44-4c35-adf5-3fffab06b644\") " pod="e2e-models-e2e-worker-w3-5a8f94/distinct-2-w3-5a8f94-kserve-ffcc6449b-hv48k"
Oct 30 21:00:53.719741 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:53.711873    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc0f5352-5c44-4c35-adf5-3fffab06b644-tmp-dir\") pod \"distinct-2-w3-5a8f94-kserve-ffcc6449b-hv48k\" (UID: \"bc0f5352-5c44-4c35-adf5-3fffab06b644\") " pod="e2e-models-e2e-worker-w3-5a8f94/distinct-2-w3-5a8f94-kserve-ffcc6449b-hv48k"
Oct 30 21:00:53.719741 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:53.711900    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc0f5352-5c44-4c35-adf5-3fffab06b644-tls-certs\") pod \"distinct-2-w3-5a8f94-kserve-ffcc6449b-hv48k\" (UID: \"bc0f5352-5c44-4c35-adf5-3fffab06b644\") " pod="e2e-models-e2e-worker-w3-5a8f94/distinct-2-w3-5a8f94-kserve-ffcc6449b-hv48k"
Oct 30 21:00:53.719741 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:53.713519    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc0f5352-5c44-4c35-adf5-3fffab06b644-home\") pod \"distinct-2-w3-5a8f94-kserve-ffcc6449b-hv48k\" (UID: \"bc0f5352-5c44-4c35-adf5-3fffab06b644\") " pod="e2e-models-e2e-worker-w3-5a8f94/distinct-2-w3-5a8f94-kserve-ffcc6449b-hv48k"
Oct 30 21:00:53.719741 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:53.713767    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc0f5352-5c44-4c35-adf5-3fffab06b644-model-cache\") pod \"distinct-2-w3-5a8f94-kserve-ffcc6449b-hv48k\" (UID: \"bc0f5352-5c44-4c35-adf5-3fffab06b644\") " pod="e2e-models-e2e-worker-w3-5a8f94/distinct-2-w3-5a8f94-kserve-ffcc6449b-hv48k"
Oct 30 21:00:53.719741 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:53.714713    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc0f5352-5c44-4c35-adf5-3fffab06b644-tmp-dir\") pod \"distinct-2-w3-5a8f94-kserve-ffcc6449b-hv48k\" (UID: \"bc0f5352-5c44-4c35-adf5-3fffab06b644\") " pod="e2e-models-e2e-worker-w3-5a8f94/distinct-2-w3-5a8f94-kserve-ffcc6449b-hv48k"
Oct 30 21:00:53.719741 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:53.718842    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc0f5352-5c44-4c35-adf5-3fffab06b644-tls-certs\") pod \"distinct-2-w3-5a8f94-kserve-ffcc6449b-hv48k\" (UID: \"bc0f5352-5c44-4c35-adf5-3fffab06b644\") " pod="e2e-models-e2e-worker-w3-5a8f94/distinct-2-w3-5a8f94-kserve-ffcc6449b-hv48k"
Oct 30 21:00:53.724346 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:53.722921    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc0f5352-5c44-4c35-adf5-3fffab06b644-dshm\") pod \"distinct-2-w3-5a8f94-kserve-ffcc6449b-hv48k\" (UID: \"bc0f5352-5c44-4c35-adf5-3fffab06b644\") " pod="e2e-models-e2e-worker-w3-5a8f94/distinct-2-w3-5a8f94-kserve-ffcc6449b-hv48k"
Oct 30 21:00:53.727128 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:53.727090    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tshfg\" (UniqueName: \"kubernetes.io/projected/bc0f5352-5c44-4c35-adf5-3fffab06b644-kube-api-access-tshfg\") pod \"distinct-2-w3-5a8f94-kserve-ffcc6449b-hv48k\" (UID: \"bc0f5352-5c44-4c35-adf5-3fffab06b644\") " pod="e2e-models-e2e-worker-w3-5a8f94/distinct-2-w3-5a8f94-kserve-ffcc6449b-hv48k"
Oct 30 21:00:53.820535 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:53.820478    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-5a8f94/distinct-2-w3-5a8f94-kserve-ffcc6449b-hv48k"
Oct 30 21:00:54.113819 ip-10-0-131-125 kubenswrapper[4521]: W1030 21:00:54.113788    4521 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbc0f5352_5c44_4c35_adf5_3fffab06b644.slice/crio-de7e2b95cd5b666bab8c0c5939d56b6fe2ee541f8412a447342eff576b5821e5 WatchSource:0}: Error finding container de7e2b95cd5b666bab8c0c5939d56b6fe2ee541f8412a447342eff576b5821e5: Status 404 returned error can't find the container with id de7e2b95cd5b666bab8c0c5939d56b6fe2ee541f8412a447342eff576b5821e5
Oct 30 21:00:54.114273 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:54.113980    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-5a8f94/distinct-2-w3-5a8f94-kserve-ffcc6449b-hv48k"]
Oct 30 21:00:54.540007 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:54.539969    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59c65aca-t465g" event={"ID":"fa6b7076-9843-483f-94ce-e61e47536580","Type":"ContainerStarted","Data":"68fe5fde7b1039f5b4d181fbca9c2514dcf4dc2d91a80f36f717a3125bc91654"}
Oct 30 21:00:54.540227 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:54.540061    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59c65aca-t465g" event={"ID":"fa6b7076-9843-483f-94ce-e61e47536580","Type":"ContainerStarted","Data":"b84a8d6a304501953525b2b89b29c2eca8b67049b2f9df6c162c0eb887993886"}
Oct 30 21:00:54.542376 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:54.542311    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-5a8f94/distinct-2-w3-5a8f94-kserve-ffcc6449b-hv48k" event={"ID":"bc0f5352-5c44-4c35-adf5-3fffab06b644","Type":"ContainerStarted","Data":"3e2e97d63e66c394818e2e06282f16a792a8be5acfb6030ebb53b47e8cd80d7d"}
Oct 30 21:00:54.542541 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:54.542387    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-5a8f94/distinct-2-w3-5a8f94-kserve-ffcc6449b-hv48k" event={"ID":"bc0f5352-5c44-4c35-adf5-3fffab06b644","Type":"ContainerStarted","Data":"de7e2b95cd5b666bab8c0c5939d56b6fe2ee541f8412a447342eff576b5821e5"}
Oct 30 21:00:54.565016 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:54.564944    4521 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59c65aca-t465g" podStartSLOduration=1.564931053 podStartE2EDuration="1.564931053s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:00:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:00:54.56044929 +0000 UTC m=+1321.906070532" watchObservedRunningTime="2026-10-30 21:00:54.564931053 +0000 UTC m=+1321.910552295"
Oct 30 21:00:54.821295 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:54.821212    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5a8f94/distinct-2-w3-5a8f94-kserve-ffcc6449b-hv48k"
Oct 30 21:00:55.372711 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:55.372646    4521 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-5a8f94/distinct-2-w3-5a8f94-kserve-ffcc6449b-hv48k" podStartSLOduration=2.372631744 podStartE2EDuration="2.372631744s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:00:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:00:54.578835227 +0000 UTC m=+1321.924456465" watchObservedRunningTime="2026-10-30 21:00:55.372631744 +0000 UTC m=+1322.718252982"
Oct 30 21:00:55.375797 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:55.375770    4521 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-w5lzs"]
Oct 30 21:00:55.379684 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:55.379664    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-w5lzs"
Oct 30 21:00:55.387771 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:55.387741    4521 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-5a8f94-openshift-default-dockercfg-vl5jm\""
Oct 30 21:00:55.392677 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:55.392652    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-w5lzs"]
Oct 30 21:00:55.454278 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:55.454245    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7f18233-f19c-4f74-a071-62d63402691c-istiod-ca-cert\") pod \"e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-w5lzs\" (UID: \"b7f18233-f19c-4f74-a071-62d63402691c\") " pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-w5lzs"
Oct 30 21:00:55.454566 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:55.454534    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7f18233-f19c-4f74-a071-62d63402691c-workload-certs\") pod \"e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-w5lzs\" (UID: \"b7f18233-f19c-4f74-a071-62d63402691c\") " pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-w5lzs"
Oct 30 21:00:55.454905 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:55.454625    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7f18233-f19c-4f74-a071-62d63402691c-istio-data\") pod \"e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-w5lzs\" (UID: \"b7f18233-f19c-4f74-a071-62d63402691c\") " pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-w5lzs"
Oct 30 21:00:55.454905 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:55.454748    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7f18233-f19c-4f74-a071-62d63402691c-workload-socket\") pod \"e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-w5lzs\" (UID: \"b7f18233-f19c-4f74-a071-62d63402691c\") " pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-w5lzs"
Oct 30 21:00:55.454905 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:55.454780    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbkf5\" (UniqueName: \"kubernetes.io/projected/b7f18233-f19c-4f74-a071-62d63402691c-kube-api-access-kbkf5\") pod \"e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-w5lzs\" (UID: \"b7f18233-f19c-4f74-a071-62d63402691c\") " pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-w5lzs"
Oct 30 21:00:55.454905 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:55.454824    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7f18233-f19c-4f74-a071-62d63402691c-istio-envoy\") pod \"e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-w5lzs\" (UID: \"b7f18233-f19c-4f74-a071-62d63402691c\") " pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-w5lzs"
Oct 30 21:00:55.455203 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:55.454944    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7f18233-f19c-4f74-a071-62d63402691c-istio-podinfo\") pod \"e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-w5lzs\" (UID: \"b7f18233-f19c-4f74-a071-62d63402691c\") " pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-w5lzs"
Oct 30 21:00:55.455203 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:55.455071    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7f18233-f19c-4f74-a071-62d63402691c-credential-socket\") pod \"e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-w5lzs\" (UID: \"b7f18233-f19c-4f74-a071-62d63402691c\") " pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-w5lzs"
Oct 30 21:00:55.455203 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:55.455099    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7f18233-f19c-4f74-a071-62d63402691c-istio-token\") pod \"e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-w5lzs\" (UID: \"b7f18233-f19c-4f74-a071-62d63402691c\") " pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-w5lzs"
Oct 30 21:00:55.504906 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:55.504874    4521 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-jjmw4"]
Oct 30 21:00:55.508923 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:55.508902    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-jjmw4"
Oct 30 21:00:55.528631 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:55.528599    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-jjmw4"]
Oct 30 21:00:55.553488 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:55.553451    4521 generic.go:356] "Generic (PLEG): container finished" podID="fa6b7076-9843-483f-94ce-e61e47536580" containerID="68fe5fde7b1039f5b4d181fbca9c2514dcf4dc2d91a80f36f717a3125bc91654" exitCode=0
Oct 30 21:00:55.553733 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:55.553709    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59c65aca-t465g" event={"ID":"fa6b7076-9843-483f-94ce-e61e47536580","Type":"ContainerDied","Data":"68fe5fde7b1039f5b4d181fbca9c2514dcf4dc2d91a80f36f717a3125bc91654"}
Oct 30 21:00:55.556535 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:55.556508    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7f18233-f19c-4f74-a071-62d63402691c-istiod-ca-cert\") pod \"e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-w5lzs\" (UID: \"b7f18233-f19c-4f74-a071-62d63402691c\") " pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-w5lzs"
Oct 30 21:00:55.556698 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:55.556543    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7f18233-f19c-4f74-a071-62d63402691c-workload-certs\") pod \"e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-w5lzs\" (UID: \"b7f18233-f19c-4f74-a071-62d63402691c\") " pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-w5lzs"
Oct 30 21:00:55.556698 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:55.556568    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7f18233-f19c-4f74-a071-62d63402691c-istio-data\") pod \"e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-w5lzs\" (UID: \"b7f18233-f19c-4f74-a071-62d63402691c\") " pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-w5lzs"
Oct 30 21:00:55.556698 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:55.556663    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7f18233-f19c-4f74-a071-62d63402691c-workload-socket\") pod \"e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-w5lzs\" (UID: \"b7f18233-f19c-4f74-a071-62d63402691c\") " pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-w5lzs"
Oct 30 21:00:55.556698 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:55.556688    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbkf5\" (UniqueName: \"kubernetes.io/projected/b7f18233-f19c-4f74-a071-62d63402691c-kube-api-access-kbkf5\") pod \"e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-w5lzs\" (UID: \"b7f18233-f19c-4f74-a071-62d63402691c\") " pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-w5lzs"
Oct 30 21:00:55.556941 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:55.556726    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7f18233-f19c-4f74-a071-62d63402691c-istio-envoy\") pod \"e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-w5lzs\" (UID: \"b7f18233-f19c-4f74-a071-62d63402691c\") " pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-w5lzs"
Oct 30 21:00:55.556941 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:55.556781    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7f18233-f19c-4f74-a071-62d63402691c-istio-podinfo\") pod \"e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-w5lzs\" (UID: \"b7f18233-f19c-4f74-a071-62d63402691c\") " pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-w5lzs"
Oct 30 21:00:55.556941 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:55.556899    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7f18233-f19c-4f74-a071-62d63402691c-credential-socket\") pod \"e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-w5lzs\" (UID: \"b7f18233-f19c-4f74-a071-62d63402691c\") " pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-w5lzs"
Oct 30 21:00:55.556941 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:55.556925    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7f18233-f19c-4f74-a071-62d63402691c-istio-token\") pod \"e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-w5lzs\" (UID: \"b7f18233-f19c-4f74-a071-62d63402691c\") " pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-w5lzs"
Oct 30 21:00:55.557340 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:55.557298    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7f18233-f19c-4f74-a071-62d63402691c-istiod-ca-cert\") pod \"e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-w5lzs\" (UID: \"b7f18233-f19c-4f74-a071-62d63402691c\") " pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-w5lzs"
Oct 30 21:00:55.557554 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:55.557529    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7f18233-f19c-4f74-a071-62d63402691c-workload-certs\") pod \"e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-w5lzs\" (UID: \"b7f18233-f19c-4f74-a071-62d63402691c\") " pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-w5lzs"
Oct 30 21:00:55.557835 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:55.557815    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7f18233-f19c-4f74-a071-62d63402691c-istio-data\") pod \"e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-w5lzs\" (UID: \"b7f18233-f19c-4f74-a071-62d63402691c\") " pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-w5lzs"
Oct 30 21:00:55.557916 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:55.557824    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7f18233-f19c-4f74-a071-62d63402691c-credential-socket\") pod \"e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-w5lzs\" (UID: \"b7f18233-f19c-4f74-a071-62d63402691c\") " pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-w5lzs"
Oct 30 21:00:55.558087 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:55.558064    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7f18233-f19c-4f74-a071-62d63402691c-workload-socket\") pod \"e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-w5lzs\" (UID: \"b7f18233-f19c-4f74-a071-62d63402691c\") " pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-w5lzs"
Oct 30 21:00:55.560317 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:55.560291    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7f18233-f19c-4f74-a071-62d63402691c-istio-envoy\") pod \"e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-w5lzs\" (UID: \"b7f18233-f19c-4f74-a071-62d63402691c\") " pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-w5lzs"
Oct 30 21:00:55.560689 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:55.560670    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7f18233-f19c-4f74-a071-62d63402691c-istio-podinfo\") pod \"e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-w5lzs\" (UID: \"b7f18233-f19c-4f74-a071-62d63402691c\") " pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-w5lzs"
Oct 30 21:00:55.569986 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:55.569956    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7f18233-f19c-4f74-a071-62d63402691c-istio-token\") pod \"e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-w5lzs\" (UID: \"b7f18233-f19c-4f74-a071-62d63402691c\") " pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-w5lzs"
Oct 30 21:00:55.570115 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:55.569961    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbkf5\" (UniqueName: \"kubernetes.io/projected/b7f18233-f19c-4f74-a071-62d63402691c-kube-api-access-kbkf5\") pod \"e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-w5lzs\" (UID: \"b7f18233-f19c-4f74-a071-62d63402691c\") " pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-w5lzs"
Oct 30 21:00:55.637850 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:55.637763    4521 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-hmwtj"]
Oct 30 21:00:55.644458 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:55.643627    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-hmwtj"
Oct 30 21:00:55.646835 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:55.646804    4521 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-p55rm"]
Oct 30 21:00:55.653162 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:55.653138    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-p55rm"
Oct 30 21:00:55.655686 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:55.655659    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-hmwtj"]
Oct 30 21:00:55.659463 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:55.659432    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d91a4f29-950e-4370-92e0-8ec493b72620-workload-socket\") pod \"e2e-worker-w0-4a0c5b-openshift-default-676865df99-jjmw4\" (UID: \"d91a4f29-950e-4370-92e0-8ec493b72620\") " pod="openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-jjmw4"
Oct 30 21:00:55.659659 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:55.659641    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d91a4f29-950e-4370-92e0-8ec493b72620-workload-certs\") pod \"e2e-worker-w0-4a0c5b-openshift-default-676865df99-jjmw4\" (UID: \"d91a4f29-950e-4370-92e0-8ec493b72620\") " pod="openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-jjmw4"
Oct 30 21:00:55.659822 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:55.659793    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d91a4f29-950e-4370-92e0-8ec493b72620-istio-token\") pod \"e2e-worker-w0-4a0c5b-openshift-default-676865df99-jjmw4\" (UID: \"d91a4f29-950e-4370-92e0-8ec493b72620\") " pod="openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-jjmw4"
Oct 30 21:00:55.659895 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:55.659870    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d91a4f29-950e-4370-92e0-8ec493b72620-istio-envoy\") pod \"e2e-worker-w0-4a0c5b-openshift-default-676865df99-jjmw4\" (UID: \"d91a4f29-950e-4370-92e0-8ec493b72620\") " pod="openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-jjmw4"
Oct 30 21:00:55.660000 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:55.659965    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d91a4f29-950e-4370-92e0-8ec493b72620-istiod-ca-cert\") pod \"e2e-worker-w0-4a0c5b-openshift-default-676865df99-jjmw4\" (UID: \"d91a4f29-950e-4370-92e0-8ec493b72620\") " pod="openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-jjmw4"
Oct 30 21:00:55.660084 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:55.660063    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqs62\" (UniqueName: \"kubernetes.io/projected/d91a4f29-950e-4370-92e0-8ec493b72620-kube-api-access-vqs62\") pod \"e2e-worker-w0-4a0c5b-openshift-default-676865df99-jjmw4\" (UID: \"d91a4f29-950e-4370-92e0-8ec493b72620\") " pod="openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-jjmw4"
Oct 30 21:00:55.660236 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:55.660205    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d91a4f29-950e-4370-92e0-8ec493b72620-istio-data\") pod \"e2e-worker-w0-4a0c5b-openshift-default-676865df99-jjmw4\" (UID: \"d91a4f29-950e-4370-92e0-8ec493b72620\") " pod="openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-jjmw4"
Oct 30 21:00:55.660399 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:55.660251    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d91a4f29-950e-4370-92e0-8ec493b72620-credential-socket\") pod \"e2e-worker-w0-4a0c5b-openshift-default-676865df99-jjmw4\" (UID: \"d91a4f29-950e-4370-92e0-8ec493b72620\") " pod="openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-jjmw4"
Oct 30 21:00:55.660399 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:55.660277    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d91a4f29-950e-4370-92e0-8ec493b72620-istio-podinfo\") pod \"e2e-worker-w0-4a0c5b-openshift-default-676865df99-jjmw4\" (UID: \"d91a4f29-950e-4370-92e0-8ec493b72620\") " pod="openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-jjmw4"
Oct 30 21:00:55.666365 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:55.666306    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-p55rm"]
Oct 30 21:00:55.694933 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:55.694899    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-w5lzs"
Oct 30 21:00:55.764350 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:55.762913    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6f47707-ed2c-4860-a13a-b1a5d9d1bbe3-istio-token\") pod \"e2e-worker-w2-32a969-openshift-default-8945d87fc-hmwtj\" (UID: \"f6f47707-ed2c-4860-a13a-b1a5d9d1bbe3\") " pod="openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-hmwtj"
Oct 30 21:00:55.764350 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:55.762945    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/baad1994-acb5-4aae-bb65-a5e8a5b8afdc-istio-envoy\") pod \"e2e-worker-w2-32a969-openshift-default-8945d87fc-p55rm\" (UID: \"baad1994-acb5-4aae-bb65-a5e8a5b8afdc\") " pod="openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-p55rm"
Oct 30 21:00:55.764350 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:55.762978    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6f47707-ed2c-4860-a13a-b1a5d9d1bbe3-workload-certs\") pod \"e2e-worker-w2-32a969-openshift-default-8945d87fc-hmwtj\" (UID: \"f6f47707-ed2c-4860-a13a-b1a5d9d1bbe3\") " pod="openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-hmwtj"
Oct 30 21:00:55.764350 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:55.762995    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/baad1994-acb5-4aae-bb65-a5e8a5b8afdc-workload-certs\") pod \"e2e-worker-w2-32a969-openshift-default-8945d87fc-p55rm\" (UID: \"baad1994-acb5-4aae-bb65-a5e8a5b8afdc\") " pod="openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-p55rm"
Oct 30 21:00:55.764350 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:55.763026    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d91a4f29-950e-4370-92e0-8ec493b72620-workload-socket\") pod \"e2e-worker-w0-4a0c5b-openshift-default-676865df99-jjmw4\" (UID: \"d91a4f29-950e-4370-92e0-8ec493b72620\") " pod="openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-jjmw4"
Oct 30 21:00:55.764350 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:55.763050    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/baad1994-acb5-4aae-bb65-a5e8a5b8afdc-istio-data\") pod \"e2e-worker-w2-32a969-openshift-default-8945d87fc-p55rm\" (UID: \"baad1994-acb5-4aae-bb65-a5e8a5b8afdc\") " pod="openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-p55rm"
Oct 30 21:00:55.764350 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:55.763072    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-288zg\" (UniqueName: \"kubernetes.io/projected/f6f47707-ed2c-4860-a13a-b1a5d9d1bbe3-kube-api-access-288zg\") pod \"e2e-worker-w2-32a969-openshift-default-8945d87fc-hmwtj\" (UID: \"f6f47707-ed2c-4860-a13a-b1a5d9d1bbe3\") " pod="openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-hmwtj"
Oct 30 21:00:55.764350 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:55.763088    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/baad1994-acb5-4aae-bb65-a5e8a5b8afdc-istiod-ca-cert\") pod \"e2e-worker-w2-32a969-openshift-default-8945d87fc-p55rm\" (UID: \"baad1994-acb5-4aae-bb65-a5e8a5b8afdc\") " pod="openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-p55rm"
Oct 30 21:00:55.764350 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:55.763106    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6f47707-ed2c-4860-a13a-b1a5d9d1bbe3-istio-data\") pod \"e2e-worker-w2-32a969-openshift-default-8945d87fc-hmwtj\" (UID: \"f6f47707-ed2c-4860-a13a-b1a5d9d1bbe3\") " pod="openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-hmwtj"
Oct 30 21:00:55.764350 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:55.763126    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d91a4f29-950e-4370-92e0-8ec493b72620-workload-certs\") pod \"e2e-worker-w0-4a0c5b-openshift-default-676865df99-jjmw4\" (UID: \"d91a4f29-950e-4370-92e0-8ec493b72620\") " pod="openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-jjmw4"
Oct 30 21:00:55.764350 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:55.763146    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d91a4f29-950e-4370-92e0-8ec493b72620-istio-token\") pod \"e2e-worker-w0-4a0c5b-openshift-default-676865df99-jjmw4\" (UID: \"d91a4f29-950e-4370-92e0-8ec493b72620\") " pod="openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-jjmw4"
Oct 30 21:00:55.764350 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:55.763164    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6f47707-ed2c-4860-a13a-b1a5d9d1bbe3-istio-podinfo\") pod \"e2e-worker-w2-32a969-openshift-default-8945d87fc-hmwtj\" (UID: \"f6f47707-ed2c-4860-a13a-b1a5d9d1bbe3\") " pod="openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-hmwtj"
Oct 30 21:00:55.764350 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:55.763190    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d91a4f29-950e-4370-92e0-8ec493b72620-istio-envoy\") pod \"e2e-worker-w0-4a0c5b-openshift-default-676865df99-jjmw4\" (UID: \"d91a4f29-950e-4370-92e0-8ec493b72620\") " pod="openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-jjmw4"
Oct 30 21:00:55.764350 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:55.763215    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtwp7\" (UniqueName: \"kubernetes.io/projected/baad1994-acb5-4aae-bb65-a5e8a5b8afdc-kube-api-access-qtwp7\") pod \"e2e-worker-w2-32a969-openshift-default-8945d87fc-p55rm\" (UID: \"baad1994-acb5-4aae-bb65-a5e8a5b8afdc\") " pod="openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-p55rm"
Oct 30 21:00:55.764350 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:55.763252    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6f47707-ed2c-4860-a13a-b1a5d9d1bbe3-workload-socket\") pod \"e2e-worker-w2-32a969-openshift-default-8945d87fc-hmwtj\" (UID: \"f6f47707-ed2c-4860-a13a-b1a5d9d1bbe3\") " pod="openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-hmwtj"
Oct 30 21:00:55.764350 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:55.763272    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d91a4f29-950e-4370-92e0-8ec493b72620-istiod-ca-cert\") pod \"e2e-worker-w0-4a0c5b-openshift-default-676865df99-jjmw4\" (UID: \"d91a4f29-950e-4370-92e0-8ec493b72620\") " pod="openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-jjmw4"
Oct 30 21:00:55.764350 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:55.763296    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqs62\" (UniqueName: \"kubernetes.io/projected/d91a4f29-950e-4370-92e0-8ec493b72620-kube-api-access-vqs62\") pod \"e2e-worker-w0-4a0c5b-openshift-default-676865df99-jjmw4\" (UID: \"d91a4f29-950e-4370-92e0-8ec493b72620\") " pod="openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-jjmw4"
Oct 30 21:00:55.764350 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:55.763350    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6f47707-ed2c-4860-a13a-b1a5d9d1bbe3-istiod-ca-cert\") pod \"e2e-worker-w2-32a969-openshift-default-8945d87fc-hmwtj\" (UID: \"f6f47707-ed2c-4860-a13a-b1a5d9d1bbe3\") " pod="openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-hmwtj"
Oct 30 21:00:55.764350 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:55.763368    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/baad1994-acb5-4aae-bb65-a5e8a5b8afdc-credential-socket\") pod \"e2e-worker-w2-32a969-openshift-default-8945d87fc-p55rm\" (UID: \"baad1994-acb5-4aae-bb65-a5e8a5b8afdc\") " pod="openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-p55rm"
Oct 30 21:00:55.764350 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:55.763399    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6f47707-ed2c-4860-a13a-b1a5d9d1bbe3-istio-envoy\") pod \"e2e-worker-w2-32a969-openshift-default-8945d87fc-hmwtj\" (UID: \"f6f47707-ed2c-4860-a13a-b1a5d9d1bbe3\") " pod="openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-hmwtj"
Oct 30 21:00:55.764350 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:55.763416    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/baad1994-acb5-4aae-bb65-a5e8a5b8afdc-workload-socket\") pod \"e2e-worker-w2-32a969-openshift-default-8945d87fc-p55rm\" (UID: \"baad1994-acb5-4aae-bb65-a5e8a5b8afdc\") " pod="openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-p55rm"
Oct 30 21:00:55.764350 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:55.763436    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d91a4f29-950e-4370-92e0-8ec493b72620-istio-data\") pod \"e2e-worker-w0-4a0c5b-openshift-default-676865df99-jjmw4\" (UID: \"d91a4f29-950e-4370-92e0-8ec493b72620\") " pod="openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-jjmw4"
Oct 30 21:00:55.764350 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:55.763467    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d91a4f29-950e-4370-92e0-8ec493b72620-credential-socket\") pod \"e2e-worker-w0-4a0c5b-openshift-default-676865df99-jjmw4\" (UID: \"d91a4f29-950e-4370-92e0-8ec493b72620\") " pod="openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-jjmw4"
Oct 30 21:00:55.764350 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:55.763482    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6f47707-ed2c-4860-a13a-b1a5d9d1bbe3-credential-socket\") pod \"e2e-worker-w2-32a969-openshift-default-8945d87fc-hmwtj\" (UID: \"f6f47707-ed2c-4860-a13a-b1a5d9d1bbe3\") " pod="openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-hmwtj"
Oct 30 21:00:55.764350 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:55.763499    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d91a4f29-950e-4370-92e0-8ec493b72620-istio-podinfo\") pod \"e2e-worker-w0-4a0c5b-openshift-default-676865df99-jjmw4\" (UID: \"d91a4f29-950e-4370-92e0-8ec493b72620\") " pod="openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-jjmw4"
Oct 30 21:00:55.764350 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:55.763517    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/baad1994-acb5-4aae-bb65-a5e8a5b8afdc-istio-token\") pod \"e2e-worker-w2-32a969-openshift-default-8945d87fc-p55rm\" (UID: \"baad1994-acb5-4aae-bb65-a5e8a5b8afdc\") " pod="openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-p55rm"
Oct 30 21:00:55.764350 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:55.763577    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/baad1994-acb5-4aae-bb65-a5e8a5b8afdc-istio-podinfo\") pod \"e2e-worker-w2-32a969-openshift-default-8945d87fc-p55rm\" (UID: \"baad1994-acb5-4aae-bb65-a5e8a5b8afdc\") " pod="openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-p55rm"
Oct 30 21:00:55.764350 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:55.763955    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d91a4f29-950e-4370-92e0-8ec493b72620-workload-socket\") pod \"e2e-worker-w0-4a0c5b-openshift-default-676865df99-jjmw4\" (UID: \"d91a4f29-950e-4370-92e0-8ec493b72620\") " pod="openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-jjmw4"
Oct 30 21:00:55.764350 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:55.764163    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d91a4f29-950e-4370-92e0-8ec493b72620-workload-certs\") pod \"e2e-worker-w0-4a0c5b-openshift-default-676865df99-jjmw4\" (UID: \"d91a4f29-950e-4370-92e0-8ec493b72620\") " pod="openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-jjmw4"
Oct 30 21:00:55.766007 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:55.765149    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d91a4f29-950e-4370-92e0-8ec493b72620-istiod-ca-cert\") pod \"e2e-worker-w0-4a0c5b-openshift-default-676865df99-jjmw4\" (UID: \"d91a4f29-950e-4370-92e0-8ec493b72620\") " pod="openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-jjmw4"
Oct 30 21:00:55.766007 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:55.765154    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d91a4f29-950e-4370-92e0-8ec493b72620-istio-data\") pod \"e2e-worker-w0-4a0c5b-openshift-default-676865df99-jjmw4\" (UID: \"d91a4f29-950e-4370-92e0-8ec493b72620\") " pod="openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-jjmw4"
Oct 30 21:00:55.766007 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:55.765416    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d91a4f29-950e-4370-92e0-8ec493b72620-credential-socket\") pod \"e2e-worker-w0-4a0c5b-openshift-default-676865df99-jjmw4\" (UID: \"d91a4f29-950e-4370-92e0-8ec493b72620\") " pod="openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-jjmw4"
Oct 30 21:00:55.766851 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:55.766827    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d91a4f29-950e-4370-92e0-8ec493b72620-istio-envoy\") pod \"e2e-worker-w0-4a0c5b-openshift-default-676865df99-jjmw4\" (UID: \"d91a4f29-950e-4370-92e0-8ec493b72620\") " pod="openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-jjmw4"
Oct 30 21:00:55.767409 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:55.767365    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d91a4f29-950e-4370-92e0-8ec493b72620-istio-podinfo\") pod \"e2e-worker-w0-4a0c5b-openshift-default-676865df99-jjmw4\" (UID: \"d91a4f29-950e-4370-92e0-8ec493b72620\") " pod="openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-jjmw4"
Oct 30 21:00:55.790024 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:55.789996    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d91a4f29-950e-4370-92e0-8ec493b72620-istio-token\") pod \"e2e-worker-w0-4a0c5b-openshift-default-676865df99-jjmw4\" (UID: \"d91a4f29-950e-4370-92e0-8ec493b72620\") " pod="openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-jjmw4"
Oct 30 21:00:55.791239 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:55.791213    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqs62\" (UniqueName: \"kubernetes.io/projected/d91a4f29-950e-4370-92e0-8ec493b72620-kube-api-access-vqs62\") pod \"e2e-worker-w0-4a0c5b-openshift-default-676865df99-jjmw4\" (UID: \"d91a4f29-950e-4370-92e0-8ec493b72620\") " pod="openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-jjmw4"
Oct 30 21:00:55.826400 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:55.826274    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-jjmw4"
Oct 30 21:00:55.867168 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:55.864770    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-288zg\" (UniqueName: \"kubernetes.io/projected/f6f47707-ed2c-4860-a13a-b1a5d9d1bbe3-kube-api-access-288zg\") pod \"e2e-worker-w2-32a969-openshift-default-8945d87fc-hmwtj\" (UID: \"f6f47707-ed2c-4860-a13a-b1a5d9d1bbe3\") " pod="openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-hmwtj"
Oct 30 21:00:55.867168 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:55.864827    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/baad1994-acb5-4aae-bb65-a5e8a5b8afdc-istiod-ca-cert\") pod \"e2e-worker-w2-32a969-openshift-default-8945d87fc-p55rm\" (UID: \"baad1994-acb5-4aae-bb65-a5e8a5b8afdc\") " pod="openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-p55rm"
Oct 30 21:00:55.867168 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:55.864858    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6f47707-ed2c-4860-a13a-b1a5d9d1bbe3-istio-data\") pod \"e2e-worker-w2-32a969-openshift-default-8945d87fc-hmwtj\" (UID: \"f6f47707-ed2c-4860-a13a-b1a5d9d1bbe3\") " pod="openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-hmwtj"
Oct 30 21:00:55.867168 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:55.864902    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6f47707-ed2c-4860-a13a-b1a5d9d1bbe3-istio-podinfo\") pod \"e2e-worker-w2-32a969-openshift-default-8945d87fc-hmwtj\" (UID: \"f6f47707-ed2c-4860-a13a-b1a5d9d1bbe3\") " pod="openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-hmwtj"
Oct 30 21:00:55.867168 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:55.864960    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtwp7\" (UniqueName: \"kubernetes.io/projected/baad1994-acb5-4aae-bb65-a5e8a5b8afdc-kube-api-access-qtwp7\") pod \"e2e-worker-w2-32a969-openshift-default-8945d87fc-p55rm\" (UID: \"baad1994-acb5-4aae-bb65-a5e8a5b8afdc\") " pod="openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-p55rm"
Oct 30 21:00:55.867168 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:55.865013    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6f47707-ed2c-4860-a13a-b1a5d9d1bbe3-workload-socket\") pod \"e2e-worker-w2-32a969-openshift-default-8945d87fc-hmwtj\" (UID: \"f6f47707-ed2c-4860-a13a-b1a5d9d1bbe3\") " pod="openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-hmwtj"
Oct 30 21:00:55.867168 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:55.865087    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6f47707-ed2c-4860-a13a-b1a5d9d1bbe3-istiod-ca-cert\") pod \"e2e-worker-w2-32a969-openshift-default-8945d87fc-hmwtj\" (UID: \"f6f47707-ed2c-4860-a13a-b1a5d9d1bbe3\") " pod="openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-hmwtj"
Oct 30 21:00:55.867168 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:55.865113    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/baad1994-acb5-4aae-bb65-a5e8a5b8afdc-credential-socket\") pod \"e2e-worker-w2-32a969-openshift-default-8945d87fc-p55rm\" (UID: \"baad1994-acb5-4aae-bb65-a5e8a5b8afdc\") " pod="openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-p55rm"
Oct 30 21:00:55.867168 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:55.865157    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6f47707-ed2c-4860-a13a-b1a5d9d1bbe3-istio-envoy\") pod \"e2e-worker-w2-32a969-openshift-default-8945d87fc-hmwtj\" (UID: \"f6f47707-ed2c-4860-a13a-b1a5d9d1bbe3\") " pod="openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-hmwtj"
Oct 30 21:00:55.867168 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:55.865182    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/baad1994-acb5-4aae-bb65-a5e8a5b8afdc-workload-socket\") pod \"e2e-worker-w2-32a969-openshift-default-8945d87fc-p55rm\" (UID: \"baad1994-acb5-4aae-bb65-a5e8a5b8afdc\") " pod="openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-p55rm"
Oct 30 21:00:55.867168 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:55.865233    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6f47707-ed2c-4860-a13a-b1a5d9d1bbe3-credential-socket\") pod \"e2e-worker-w2-32a969-openshift-default-8945d87fc-hmwtj\" (UID: \"f6f47707-ed2c-4860-a13a-b1a5d9d1bbe3\") " pod="openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-hmwtj"
Oct 30 21:00:55.867168 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:55.865262    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/baad1994-acb5-4aae-bb65-a5e8a5b8afdc-istio-token\") pod \"e2e-worker-w2-32a969-openshift-default-8945d87fc-p55rm\" (UID: \"baad1994-acb5-4aae-bb65-a5e8a5b8afdc\") " pod="openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-p55rm"
Oct 30 21:00:55.867168 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:55.865387    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/baad1994-acb5-4aae-bb65-a5e8a5b8afdc-istio-podinfo\") pod \"e2e-worker-w2-32a969-openshift-default-8945d87fc-p55rm\" (UID: \"baad1994-acb5-4aae-bb65-a5e8a5b8afdc\") " pod="openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-p55rm"
Oct 30 21:00:55.867168 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:55.865417    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6f47707-ed2c-4860-a13a-b1a5d9d1bbe3-istio-token\") pod \"e2e-worker-w2-32a969-openshift-default-8945d87fc-hmwtj\" (UID: \"f6f47707-ed2c-4860-a13a-b1a5d9d1bbe3\") " pod="openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-hmwtj"
Oct 30 21:00:55.867168 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:55.865440    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/baad1994-acb5-4aae-bb65-a5e8a5b8afdc-istio-envoy\") pod \"e2e-worker-w2-32a969-openshift-default-8945d87fc-p55rm\" (UID: \"baad1994-acb5-4aae-bb65-a5e8a5b8afdc\") " pod="openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-p55rm"
Oct 30 21:00:55.867168 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:55.865489    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6f47707-ed2c-4860-a13a-b1a5d9d1bbe3-workload-certs\") pod \"e2e-worker-w2-32a969-openshift-default-8945d87fc-hmwtj\" (UID: \"f6f47707-ed2c-4860-a13a-b1a5d9d1bbe3\") " pod="openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-hmwtj"
Oct 30 21:00:55.867168 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:55.865514    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/baad1994-acb5-4aae-bb65-a5e8a5b8afdc-workload-certs\") pod \"e2e-worker-w2-32a969-openshift-default-8945d87fc-p55rm\" (UID: \"baad1994-acb5-4aae-bb65-a5e8a5b8afdc\") " pod="openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-p55rm"
Oct 30 21:00:55.867168 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:55.865562    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/baad1994-acb5-4aae-bb65-a5e8a5b8afdc-istio-data\") pod \"e2e-worker-w2-32a969-openshift-default-8945d87fc-p55rm\" (UID: \"baad1994-acb5-4aae-bb65-a5e8a5b8afdc\") " pod="openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-p55rm"
Oct 30 21:00:55.867168 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:55.866070    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/baad1994-acb5-4aae-bb65-a5e8a5b8afdc-istio-data\") pod \"e2e-worker-w2-32a969-openshift-default-8945d87fc-p55rm\" (UID: \"baad1994-acb5-4aae-bb65-a5e8a5b8afdc\") " pod="openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-p55rm"
Oct 30 21:00:55.867168 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:55.866815    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/baad1994-acb5-4aae-bb65-a5e8a5b8afdc-workload-socket\") pod \"e2e-worker-w2-32a969-openshift-default-8945d87fc-p55rm\" (UID: \"baad1994-acb5-4aae-bb65-a5e8a5b8afdc\") " pod="openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-p55rm"
Oct 30 21:00:55.867168 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:55.867080    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6f47707-ed2c-4860-a13a-b1a5d9d1bbe3-credential-socket\") pod \"e2e-worker-w2-32a969-openshift-default-8945d87fc-hmwtj\" (UID: \"f6f47707-ed2c-4860-a13a-b1a5d9d1bbe3\") " pod="openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-hmwtj"
Oct 30 21:00:55.868445 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:55.867252    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6f47707-ed2c-4860-a13a-b1a5d9d1bbe3-istiod-ca-cert\") pod \"e2e-worker-w2-32a969-openshift-default-8945d87fc-hmwtj\" (UID: \"f6f47707-ed2c-4860-a13a-b1a5d9d1bbe3\") " pod="openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-hmwtj"
Oct 30 21:00:55.868445 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:55.867280    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6f47707-ed2c-4860-a13a-b1a5d9d1bbe3-workload-socket\") pod \"e2e-worker-w2-32a969-openshift-default-8945d87fc-hmwtj\" (UID: \"f6f47707-ed2c-4860-a13a-b1a5d9d1bbe3\") " pod="openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-hmwtj"
Oct 30 21:00:55.868445 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:55.867587    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6f47707-ed2c-4860-a13a-b1a5d9d1bbe3-workload-certs\") pod \"e2e-worker-w2-32a969-openshift-default-8945d87fc-hmwtj\" (UID: \"f6f47707-ed2c-4860-a13a-b1a5d9d1bbe3\") " pod="openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-hmwtj"
Oct 30 21:00:55.868445 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:55.867590    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/baad1994-acb5-4aae-bb65-a5e8a5b8afdc-istiod-ca-cert\") pod \"e2e-worker-w2-32a969-openshift-default-8945d87fc-p55rm\" (UID: \"baad1994-acb5-4aae-bb65-a5e8a5b8afdc\") " pod="openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-p55rm"
Oct 30 21:00:55.868445 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:55.867703    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6f47707-ed2c-4860-a13a-b1a5d9d1bbe3-istio-data\") pod \"e2e-worker-w2-32a969-openshift-default-8945d87fc-hmwtj\" (UID: \"f6f47707-ed2c-4860-a13a-b1a5d9d1bbe3\") " pod="openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-hmwtj"
Oct 30 21:00:55.868445 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:55.867985    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/baad1994-acb5-4aae-bb65-a5e8a5b8afdc-credential-socket\") pod \"e2e-worker-w2-32a969-openshift-default-8945d87fc-p55rm\" (UID: \"baad1994-acb5-4aae-bb65-a5e8a5b8afdc\") " pod="openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-p55rm"
Oct 30 21:00:55.868445 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:55.868124    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/baad1994-acb5-4aae-bb65-a5e8a5b8afdc-workload-certs\") pod \"e2e-worker-w2-32a969-openshift-default-8945d87fc-p55rm\" (UID: \"baad1994-acb5-4aae-bb65-a5e8a5b8afdc\") " pod="openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-p55rm"
Oct 30 21:00:55.869430 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:55.869405    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6f47707-ed2c-4860-a13a-b1a5d9d1bbe3-istio-envoy\") pod \"e2e-worker-w2-32a969-openshift-default-8945d87fc-hmwtj\" (UID: \"f6f47707-ed2c-4860-a13a-b1a5d9d1bbe3\") " pod="openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-hmwtj"
Oct 30 21:00:55.869947 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:55.869929    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/baad1994-acb5-4aae-bb65-a5e8a5b8afdc-istio-envoy\") pod \"e2e-worker-w2-32a969-openshift-default-8945d87fc-p55rm\" (UID: \"baad1994-acb5-4aae-bb65-a5e8a5b8afdc\") " pod="openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-p55rm"
Oct 30 21:00:55.870354 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:55.870335    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6f47707-ed2c-4860-a13a-b1a5d9d1bbe3-istio-podinfo\") pod \"e2e-worker-w2-32a969-openshift-default-8945d87fc-hmwtj\" (UID: \"f6f47707-ed2c-4860-a13a-b1a5d9d1bbe3\") " pod="openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-hmwtj"
Oct 30 21:00:55.870658 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:55.870640    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/baad1994-acb5-4aae-bb65-a5e8a5b8afdc-istio-podinfo\") pod \"e2e-worker-w2-32a969-openshift-default-8945d87fc-p55rm\" (UID: \"baad1994-acb5-4aae-bb65-a5e8a5b8afdc\") " pod="openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-p55rm"
Oct 30 21:00:55.892195 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:55.889877    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/baad1994-acb5-4aae-bb65-a5e8a5b8afdc-istio-token\") pod \"e2e-worker-w2-32a969-openshift-default-8945d87fc-p55rm\" (UID: \"baad1994-acb5-4aae-bb65-a5e8a5b8afdc\") " pod="openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-p55rm"
Oct 30 21:00:55.892195 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:55.890947    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-288zg\" (UniqueName: \"kubernetes.io/projected/f6f47707-ed2c-4860-a13a-b1a5d9d1bbe3-kube-api-access-288zg\") pod \"e2e-worker-w2-32a969-openshift-default-8945d87fc-hmwtj\" (UID: \"f6f47707-ed2c-4860-a13a-b1a5d9d1bbe3\") " pod="openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-hmwtj"
Oct 30 21:00:55.892195 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:55.891816    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6f47707-ed2c-4860-a13a-b1a5d9d1bbe3-istio-token\") pod \"e2e-worker-w2-32a969-openshift-default-8945d87fc-hmwtj\" (UID: \"f6f47707-ed2c-4860-a13a-b1a5d9d1bbe3\") " pod="openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-hmwtj"
Oct 30 21:00:55.892195 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:55.892149    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtwp7\" (UniqueName: \"kubernetes.io/projected/baad1994-acb5-4aae-bb65-a5e8a5b8afdc-kube-api-access-qtwp7\") pod \"e2e-worker-w2-32a969-openshift-default-8945d87fc-p55rm\" (UID: \"baad1994-acb5-4aae-bb65-a5e8a5b8afdc\") " pod="openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-p55rm"
Oct 30 21:00:55.975781 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:55.975749    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-hmwtj"
Oct 30 21:00:55.986119 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:55.986088    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-p55rm"
Oct 30 21:00:55.994866 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:55.994841    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-w5lzs"]
Oct 30 21:00:55.996568 ip-10-0-131-125 kubenswrapper[4521]: W1030 21:00:55.996545    4521 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7f18233_f19c_4f74_a071_62d63402691c.slice/crio-781e27a63bceb65a8f8fe6fd87bd3a8f35913ae7e2e9e996a42424b6c18300d9 WatchSource:0}: Error finding container 781e27a63bceb65a8f8fe6fd87bd3a8f35913ae7e2e9e996a42424b6c18300d9: Status 404 returned error can't find the container with id 781e27a63bceb65a8f8fe6fd87bd3a8f35913ae7e2e9e996a42424b6c18300d9
Oct 30 21:00:56.000991 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:56.000951    4521 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:00:56.001070 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:56.001043    4521 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:00:56.001118 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:56.001093    4521 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:00:56.125532 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:56.124873    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-jjmw4"]
Oct 30 21:00:56.134301 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:56.134268    4521 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:00:56.134429 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:56.134386    4521 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:00:56.134497 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:56.134433    4521 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:00:56.321266 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:56.321183    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-hmwtj"]
Oct 30 21:00:56.329391 ip-10-0-131-125 kubenswrapper[4521]: W1030 21:00:56.329270    4521 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6f47707_ed2c_4860_a13a_b1a5d9d1bbe3.slice/crio-b31b2b9d5bb2a2d3056bf95aafc88f4cbd316eb4b9143c6faf29d592f54df223 WatchSource:0}: Error finding container b31b2b9d5bb2a2d3056bf95aafc88f4cbd316eb4b9143c6faf29d592f54df223: Status 404 returned error can't find the container with id b31b2b9d5bb2a2d3056bf95aafc88f4cbd316eb4b9143c6faf29d592f54df223
Oct 30 21:00:56.332930 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:56.332891    4521 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:00:56.333011 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:56.332980    4521 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:00:56.333078 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:56.333024    4521 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:00:56.354966 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:56.353807    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-p55rm"]
Oct 30 21:00:56.357672 ip-10-0-131-125 kubenswrapper[4521]: W1030 21:00:56.357633    4521 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbaad1994_acb5_4aae_bb65_a5e8a5b8afdc.slice/crio-150b3e052e1edf9d3da1f6bec81a4b61348e9462093242ccf0b16d35cefd0589 WatchSource:0}: Error finding container 150b3e052e1edf9d3da1f6bec81a4b61348e9462093242ccf0b16d35cefd0589: Status 404 returned error can't find the container with id 150b3e052e1edf9d3da1f6bec81a4b61348e9462093242ccf0b16d35cefd0589
Oct 30 21:00:56.368623 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:56.368576    4521 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:00:56.368741 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:56.368677    4521 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:00:56.368741 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:56.368717    4521 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:00:56.568716 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:56.567762    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-jjmw4" event={"ID":"d91a4f29-950e-4370-92e0-8ec493b72620","Type":"ContainerStarted","Data":"949c736f77bbe18f449157304ec69d5f8dc30d34424b7bec3a13bef4c4ece749"}
Oct 30 21:00:56.568716 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:56.567819    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-jjmw4" event={"ID":"d91a4f29-950e-4370-92e0-8ec493b72620","Type":"ContainerStarted","Data":"4061230f09c87f7aff0f0c24cd4d4ec270b1e1c6842669b401905661bb47cd7c"}
Oct 30 21:00:56.571388 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:56.570759    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-p55rm" event={"ID":"baad1994-acb5-4aae-bb65-a5e8a5b8afdc","Type":"ContainerStarted","Data":"792caf04aadf9b69e35ec7d46269a59947e1107749f2465ea6a82633fae13abd"}
Oct 30 21:00:56.571388 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:56.570799    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-p55rm" event={"ID":"baad1994-acb5-4aae-bb65-a5e8a5b8afdc","Type":"ContainerStarted","Data":"150b3e052e1edf9d3da1f6bec81a4b61348e9462093242ccf0b16d35cefd0589"}
Oct 30 21:00:56.574130 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:56.573646    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-hmwtj" event={"ID":"f6f47707-ed2c-4860-a13a-b1a5d9d1bbe3","Type":"ContainerStarted","Data":"8b8bd567c67d21ef122a8fd6141d1c042d7bfca248ef665aa84d0e993110b661"}
Oct 30 21:00:56.574130 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:56.573679    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-hmwtj" event={"ID":"f6f47707-ed2c-4860-a13a-b1a5d9d1bbe3","Type":"ContainerStarted","Data":"b31b2b9d5bb2a2d3056bf95aafc88f4cbd316eb4b9143c6faf29d592f54df223"}
Oct 30 21:00:56.576714 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:56.576589    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-w5lzs" event={"ID":"b7f18233-f19c-4f74-a071-62d63402691c","Type":"ContainerStarted","Data":"cf3a9a5ec8ba245d0d64dedab1e441bbfab2abe751df9baeae4e64280ecf8c4f"}
Oct 30 21:00:56.576714 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:56.576633    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-w5lzs" event={"ID":"b7f18233-f19c-4f74-a071-62d63402691c","Type":"ContainerStarted","Data":"781e27a63bceb65a8f8fe6fd87bd3a8f35913ae7e2e9e996a42424b6c18300d9"}
Oct 30 21:00:56.602527 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:56.602470    4521 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-jjmw4" podStartSLOduration=1.60245375 podStartE2EDuration="1.60245375s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:00:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:00:56.59563036 +0000 UTC m=+1323.941251599" watchObservedRunningTime="2026-10-30 21:00:56.60245375 +0000 UTC m=+1323.948075018"
Oct 30 21:00:56.663800 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:56.663688    4521 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-p55rm" podStartSLOduration=1.663667893 podStartE2EDuration="1.663667893s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:00:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:00:56.628070882 +0000 UTC m=+1323.973692120" watchObservedRunningTime="2026-10-30 21:00:56.663667893 +0000 UTC m=+1324.009289132"
Oct 30 21:00:56.695608 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:56.695566    4521 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-w5lzs"
Oct 30 21:00:56.695829 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:56.695582    4521 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-hmwtj" podStartSLOduration=1.695568685 podStartE2EDuration="1.695568685s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:00:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:00:56.693356388 +0000 UTC m=+1324.038977619" watchObservedRunningTime="2026-10-30 21:00:56.695568685 +0000 UTC m=+1324.041189922"
Oct 30 21:00:56.697507 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:56.697461    4521 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-w5lzs" podStartSLOduration=1.69744546 podStartE2EDuration="1.69744546s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:00:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:00:56.664393776 +0000 UTC m=+1324.010015015" watchObservedRunningTime="2026-10-30 21:00:56.69744546 +0000 UTC m=+1324.043066705"
Oct 30 21:00:56.829139 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:56.828588    4521 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-jjmw4"
Oct 30 21:00:56.925526 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:56.924992    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59c65aca-t465g"
Oct 30 21:00:56.986371 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:56.983392    4521 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-hmwtj"
Oct 30 21:00:56.987063 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:56.986985    4521 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-p55rm"
Oct 30 21:00:57.108253 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:57.108037    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa6b7076-9843-483f-94ce-e61e47536580-kube-api-access-xrkwm\" (UniqueName: \"kubernetes.io/projected/fa6b7076-9843-483f-94ce-e61e47536580-kube-api-access-xrkwm\") pod \"fa6b7076-9843-483f-94ce-e61e47536580\" (UID: \"fa6b7076-9843-483f-94ce-e61e47536580\") "
Oct 30 21:00:57.109275 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:57.109241    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fa6b7076-9843-483f-94ce-e61e47536580-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa6b7076-9843-483f-94ce-e61e47536580-maas-api-xks-ca\") pod \"fa6b7076-9843-483f-94ce-e61e47536580\" (UID: \"fa6b7076-9843-483f-94ce-e61e47536580\") "
Oct 30 21:00:57.109552 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:57.109532    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa6b7076-9843-483f-94ce-e61e47536580-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa6b7076-9843-483f-94ce-e61e47536580-maas-api-openshift-ca\") pod \"fa6b7076-9843-483f-94ce-e61e47536580\" (UID: \"fa6b7076-9843-483f-94ce-e61e47536580\") "
Oct 30 21:00:57.129405 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:57.129363    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa6b7076-9843-483f-94ce-e61e47536580-maas-api-openshift-ca" pod "fa6b7076-9843-483f-94ce-e61e47536580" (UID: "fa6b7076-9843-483f-94ce-e61e47536580"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:00:57.207565 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:57.207473    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa6b7076-9843-483f-94ce-e61e47536580-maas-api-xks-ca" pod "fa6b7076-9843-483f-94ce-e61e47536580" (UID: "fa6b7076-9843-483f-94ce-e61e47536580"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:00:57.214362 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:57.213496    4521 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa6b7076-9843-483f-94ce-e61e47536580-maas-api-xks-ca\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:00:57.214362 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:57.213540    4521 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa6b7076-9843-483f-94ce-e61e47536580-maas-api-openshift-ca\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:00:57.214362 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:57.213685    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa6b7076-9843-483f-94ce-e61e47536580-kube-api-access-xrkwm" pod "fa6b7076-9843-483f-94ce-e61e47536580" (UID: "fa6b7076-9843-483f-94ce-e61e47536580"). InnerVolumeSpecName "kube-api-access-xrkwm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:00:57.319338 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:57.318576    4521 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrkwm\" (UniqueName: \"kubernetes.io/projected/fa6b7076-9843-483f-94ce-e61e47536580-kube-api-access-xrkwm\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:00:57.679886 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:57.679856    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59c65aca-t465g"
Oct 30 21:00:57.729352 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:57.711506    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59c65aca-t465g" event={"ID":"fa6b7076-9843-483f-94ce-e61e47536580","Type":"ContainerDied","Data":"b84a8d6a304501953525b2b89b29c2eca8b67049b2f9df6c162c0eb887993886"}
Oct 30 21:00:57.729352 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:57.711572    4521 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b84a8d6a304501953525b2b89b29c2eca8b67049b2f9df6c162c0eb887993886"
Oct 30 21:00:57.730975 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:57.730944    4521 patch_prober.go:29] interesting pod/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-w5lzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Oct 30 21:00:57.731254 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:57.731226    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-w5lzs" podUID="b7f18233-f19c-4f74-a071-62d63402691c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Oct 30 21:00:57.841348 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:57.832212    4521 patch_prober.go:29] interesting pod/e2e-worker-w0-4a0c5b-openshift-default-676865df99-jjmw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 30 21:00:57.841348 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:57.832296    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-jjmw4" podUID="d91a4f29-950e-4370-92e0-8ec493b72620" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 30 21:00:57.987961 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:57.987919    4521 patch_prober.go:29] interesting pod/e2e-worker-w2-32a969-openshift-default-8945d87fc-hmwtj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 30 21:00:57.988236 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:57.988201    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-hmwtj" podUID="f6f47707-ed2c-4860-a13a-b1a5d9d1bbe3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 30 21:00:57.998802 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:57.994105    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-32a969/simulator-w2-32a969-kserve-6cc8dfc44c-dwhlt" podUID="62a6adf7-d17a-4fec-a707-a2d549b67753" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:00:58.027347 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:58.025419    4521 patch_prober.go:29] interesting pod/e2e-worker-w2-32a969-openshift-default-8945d87fc-p55rm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Oct 30 21:00:58.027347 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:58.025483    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-p55rm" podUID="baad1994-acb5-4aae-bb65-a5e8a5b8afdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Oct 30 21:00:58.072349 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:58.071339    4521 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-32a969/distinct-w2-32a969-kserve-7f798b664f-jdh82" podUID="887f1d27-bf24-4b57-b6e4-8fb095df7a50" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:00:58.472349 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:58.449495    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-32a969/distinct-2-w2-32a969-kserve-5c7845448b-vt5gw" podUID="d430ed01-65d1-43a6-a0c4-ab82212376a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded"
Oct 30 21:00:58.713912 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:58.713876    4521 patch_prober.go:29] interesting pod/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-w5lzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Oct 30 21:00:58.714524 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:58.714490    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-w5lzs" podUID="b7f18233-f19c-4f74-a071-62d63402691c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Oct 30 21:00:58.730580 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:58.730439    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-32a969/distinct-w2-32a969-kserve-7f798b664f-jdh82" podUID="887f1d27-bf24-4b57-b6e4-8fb095df7a50" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:00:58.783774 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:58.783733    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c2d1eb/premium-w4-c2d1eb-kserve-8696fbd9b9-mxqhm" podUID="4bb0f898-b77f-419d-bff4-6413e82ef51d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:00:58.830356 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:58.827714    4521 patch_prober.go:29] interesting pod/e2e-worker-w0-4a0c5b-openshift-default-676865df99-jjmw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 30 21:00:58.830356 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:58.827783    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-jjmw4" podUID="d91a4f29-950e-4370-92e0-8ec493b72620" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 30 21:00:58.977359 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:58.976363    4521 patch_prober.go:29] interesting pod/e2e-worker-w2-32a969-openshift-default-8945d87fc-hmwtj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 30 21:00:58.977359 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:58.976427    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-hmwtj" podUID="f6f47707-ed2c-4860-a13a-b1a5d9d1bbe3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 30 21:00:58.986004 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:58.984503    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-39fb96/premium-w6-39fb96-kserve-b9cf7f75c-96blx" podUID="18ca0a94-c322-41fa-9fd9-0f14419b5075" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:00:58.994771 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:58.994734    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-32a969/simulator-w2-32a969-kserve-6cc8dfc44c-dwhlt" podUID="62a6adf7-d17a-4fec-a707-a2d549b67753" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Oct 30 21:00:58.995273 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:58.995244    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a0c5b/premium-w0-4a0c5b-kserve-85dcf89f57-mscdn" podUID="00e42758-77ce-4663-adfa-2202c0aae24a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:00:58.995758 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:58.995739    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-32a969/simulator-w2-32a969-kserve-6cc8dfc44c-dwhlt"
Oct 30 21:00:59.005117 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:59.003859    4521 patch_prober.go:29] interesting pod/e2e-worker-w2-32a969-openshift-default-8945d87fc-p55rm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Oct 30 21:00:59.005117 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:59.003905    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-p55rm" podUID="baad1994-acb5-4aae-bb65-a5e8a5b8afdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Oct 30 21:00:59.034584 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:59.027411    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-65968b/premium-w5-65968b-kserve-67977485f7-c9jpv" podUID="92b2dd65-8a83-436d-afa8-50e5f0d11fa0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:00:59.076350 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:59.070777    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a8f94/simulator-w3-5a8f94-kserve-77cc88cc89-n4sl6" podUID="1fa6900c-eee2-4f00-b0ec-248def48dfc4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:00:59.098471 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:59.098423    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c4b379/premium-w1-c4b379-kserve-575f896d49-2n6gz" podUID="2aac3e8e-d631-4b9c-bdbb-3b4b1df133a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:00:59.099379 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:59.099092    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a8f94/premium-w3-5a8f94-kserve-bb96b6848-5hk8h" podUID="24ccc37d-027a-4097-8da5-04df417aef94" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Oct 30 21:00:59.154565 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:59.138255    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a8f94/distinct-w3-5a8f94-kserve-5f8b544d59-mf95j" podUID="b8ac1cca-4ebd-474c-b32f-a3595a4cd847" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:00:59.462350 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:59.453412    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-32a969/distinct-2-w2-32a969-kserve-5c7845448b-vt5gw" podUID="d430ed01-65d1-43a6-a0c4-ab82212376a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:00:59.462350 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:59.454079    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-32a969/distinct-2-w2-32a969-kserve-5c7845448b-vt5gw"
Oct 30 21:00:59.708351 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:59.705012    4521 patch_prober.go:29] interesting pod/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-w5lzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Oct 30 21:00:59.708351 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:59.705074    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-w5lzs" podUID="b7f18233-f19c-4f74-a071-62d63402691c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Oct 30 21:00:59.801405 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:59.769238    4521 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-847c3a60-bzqgs"]
Oct 30 21:00:59.802801 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:59.802771    4521 cpu_manager.go:414] "containerMap: removing container" podUID="fa6b7076-9843-483f-94ce-e61e47536580" containerName="revoke-keys"
Oct 30 21:00:59.802977 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:59.802963    4521 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa6b7076-9843-483f-94ce-e61e47536580" containerName="revoke-keys"
Oct 30 21:00:59.803240 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:59.803226    4521 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa6b7076-9843-483f-94ce-e61e47536580" containerName="revoke-keys"
Oct 30 21:00:59.805739 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:59.785238    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c2d1eb/premium-w4-c2d1eb-kserve-8696fbd9b9-mxqhm" podUID="4bb0f898-b77f-419d-bff4-6413e82ef51d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:00:59.807695 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:59.807668    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c2d1eb/premium-w4-c2d1eb-kserve-8696fbd9b9-mxqhm"
Oct 30 21:00:59.807796 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:59.807730    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-847c3a60-bzqgs"]
Oct 30 21:00:59.808208 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:59.808186    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-847c3a60-bzqgs"
Oct 30 21:00:59.814896 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:59.814875    4521 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:00:59.818049 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:59.818029    4521 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c45pw\""
Oct 30 21:00:59.827607 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:59.827584    4521 patch_prober.go:29] interesting pod/e2e-worker-w0-4a0c5b-openshift-default-676865df99-jjmw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 30 21:00:59.843773 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:59.843738    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-jjmw4" podUID="d91a4f29-950e-4370-92e0-8ec493b72620" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 30 21:00:59.962901 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:59.962861    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/655b039a-00a8-40fd-baaa-e58a80149a06-maas-api-xks-ca\") pod \"maas-api-revoke-sub-847c3a60-bzqgs\" (UID: \"655b039a-00a8-40fd-baaa-e58a80149a06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-847c3a60-bzqgs"
Oct 30 21:00:59.963261 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:59.963231    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzj5q\" (UniqueName: \"kubernetes.io/projected/655b039a-00a8-40fd-baaa-e58a80149a06-kube-api-access-wzj5q\") pod \"maas-api-revoke-sub-847c3a60-bzqgs\" (UID: \"655b039a-00a8-40fd-baaa-e58a80149a06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-847c3a60-bzqgs"
Oct 30 21:00:59.963488 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:00:59.963466    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/655b039a-00a8-40fd-baaa-e58a80149a06-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-847c3a60-bzqgs\" (UID: \"655b039a-00a8-40fd-baaa-e58a80149a06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-847c3a60-bzqgs"
Oct 30 21:01:00.102410 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:00.063537    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-65968b/premium-w5-65968b-kserve-67977485f7-c9jpv" podUID="92b2dd65-8a83-436d-afa8-50e5f0d11fa0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:01:00.110785 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:00.065015    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/655b039a-00a8-40fd-baaa-e58a80149a06-maas-api-xks-ca\") pod \"maas-api-revoke-sub-847c3a60-bzqgs\" (UID: \"655b039a-00a8-40fd-baaa-e58a80149a06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-847c3a60-bzqgs"
Oct 30 21:01:00.122554 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:00.122506    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzj5q\" (UniqueName: \"kubernetes.io/projected/655b039a-00a8-40fd-baaa-e58a80149a06-kube-api-access-wzj5q\") pod \"maas-api-revoke-sub-847c3a60-bzqgs\" (UID: \"655b039a-00a8-40fd-baaa-e58a80149a06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-847c3a60-bzqgs"
Oct 30 21:01:00.147345 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:00.140640    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/655b039a-00a8-40fd-baaa-e58a80149a06-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-847c3a60-bzqgs\" (UID: \"655b039a-00a8-40fd-baaa-e58a80149a06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-847c3a60-bzqgs"
Oct 30 21:01:00.147345 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:00.141674    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/655b039a-00a8-40fd-baaa-e58a80149a06-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-847c3a60-bzqgs\" (UID: \"655b039a-00a8-40fd-baaa-e58a80149a06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-847c3a60-bzqgs"
Oct 30 21:01:00.147345 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:00.141977    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-65968b/premium-w5-65968b-kserve-67977485f7-c9jpv"
Oct 30 21:01:00.147345 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:00.078653    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a8f94/simulator-w3-5a8f94-kserve-77cc88cc89-n4sl6" podUID="1fa6900c-eee2-4f00-b0ec-248def48dfc4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:01:00.147345 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:00.079357    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-39fb96/premium-w6-39fb96-kserve-b9cf7f75c-96blx" podUID="18ca0a94-c322-41fa-9fd9-0f14419b5075" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:01:00.147345 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:00.143824    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5a8f94/simulator-w3-5a8f94-kserve-77cc88cc89-n4sl6"
Oct 30 21:01:00.147345 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:00.143874    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-39fb96/premium-w6-39fb96-kserve-b9cf7f75c-96blx"
Oct 30 21:01:00.283349 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:00.282399    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/655b039a-00a8-40fd-baaa-e58a80149a06-maas-api-xks-ca\") pod \"maas-api-revoke-sub-847c3a60-bzqgs\" (UID: \"655b039a-00a8-40fd-baaa-e58a80149a06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-847c3a60-bzqgs"
Oct 30 21:01:00.326351 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:00.320399    4521 patch_prober.go:29] interesting pod/e2e-worker-w2-32a969-openshift-default-8945d87fc-hmwtj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 30 21:01:00.326351 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:00.320474    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-hmwtj" podUID="f6f47707-ed2c-4860-a13a-b1a5d9d1bbe3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 30 21:01:00.385412 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:00.376630    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a8f94/premium-w3-5a8f94-kserve-bb96b6848-5hk8h" podUID="24ccc37d-027a-4097-8da5-04df417aef94" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Oct 30 21:01:00.419936 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:00.391579    4521 patch_prober.go:29] interesting pod/e2e-worker-w2-32a969-openshift-default-8945d87fc-p55rm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Oct 30 21:01:00.419936 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:00.391635    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-p55rm" podUID="baad1994-acb5-4aae-bb65-a5e8a5b8afdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Oct 30 21:01:00.419936 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:00.392608    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5a8f94/premium-w3-5a8f94-kserve-bb96b6848-5hk8h"
Oct 30 21:01:00.421651 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:00.421615    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-32a969/simulator-w2-32a969-kserve-6cc8dfc44c-dwhlt" podUID="62a6adf7-d17a-4fec-a707-a2d549b67753" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Oct 30 21:01:00.422178 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:00.422149    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a8f94/distinct-w3-5a8f94-kserve-5f8b544d59-mf95j" podUID="b8ac1cca-4ebd-474c-b32f-a3595a4cd847" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded"
Oct 30 21:01:00.524670 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:00.472446    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-32a969/distinct-2-w2-32a969-kserve-5c7845448b-vt5gw" podUID="d430ed01-65d1-43a6-a0c4-ab82212376a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:01:00.613363 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:00.588769    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a0c5b/premium-w0-4a0c5b-kserve-85dcf89f57-mscdn" podUID="00e42758-77ce-4663-adfa-2202c0aae24a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:01:00.613363 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:00.591886    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzj5q\" (UniqueName: \"kubernetes.io/projected/655b039a-00a8-40fd-baaa-e58a80149a06-kube-api-access-wzj5q\") pod \"maas-api-revoke-sub-847c3a60-bzqgs\" (UID: \"655b039a-00a8-40fd-baaa-e58a80149a06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-847c3a60-bzqgs"
Oct 30 21:01:00.845703 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:00.692016    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5a8f94/distinct-w3-5a8f94-kserve-5f8b544d59-mf95j"
Oct 30 21:01:00.845703 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:00.692084    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4a0c5b/premium-w0-4a0c5b-kserve-85dcf89f57-mscdn"
Oct 30 21:01:00.845703 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:00.692126    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c2d1eb/premium-w4-c2d1eb-kserve-8696fbd9b9-mxqhm"
Oct 30 21:01:00.845703 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:00.734839    4521 patch_prober.go:29] interesting pod/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-w5lzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Oct 30 21:01:00.845703 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:00.734908    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-w5lzs" podUID="b7f18233-f19c-4f74-a071-62d63402691c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Oct 30 21:01:00.845703 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:00.735306    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-32a969/distinct-w2-32a969-kserve-7f798b664f-jdh82" podUID="887f1d27-bf24-4b57-b6e4-8fb095df7a50" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded"
Oct 30 21:01:00.845703 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:00.771841    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-847c3a60-bzqgs"
Oct 30 21:01:00.903352 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:00.901475    4521 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-32a969/distinct-2-w2-32a969-kserve-5c7845448b-vt5gw" podUID="d430ed01-65d1-43a6-a0c4-ab82212376a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:01:00.914447 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:00.912075    4521 patch_prober.go:29] interesting pod/e2e-worker-w0-4a0c5b-openshift-default-676865df99-jjmw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 30 21:01:00.914447 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:00.912140    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-jjmw4" podUID="d91a4f29-950e-4370-92e0-8ec493b72620" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 30 21:01:01.033088 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:01.026053    4521 patch_prober.go:29] interesting pod/e2e-worker-w2-32a969-openshift-default-8945d87fc-hmwtj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 30 21:01:01.033088 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:01.026113    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-hmwtj" podUID="f6f47707-ed2c-4860-a13a-b1a5d9d1bbe3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 30 21:01:01.041304 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:01.041257    4521 patch_prober.go:29] interesting pod/e2e-worker-w2-32a969-openshift-default-8945d87fc-p55rm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Oct 30 21:01:01.041727 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:01.041686    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-p55rm" podUID="baad1994-acb5-4aae-bb65-a5e8a5b8afdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Oct 30 21:01:01.115114 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:01.113755    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-65968b/premium-w5-65968b-kserve-67977485f7-c9jpv" podUID="92b2dd65-8a83-436d-afa8-50e5f0d11fa0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:01:01.170671 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:01.170633    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5a8f94/simulator-w3-5a8f94-kserve-77cc88cc89-n4sl6"
Oct 30 21:01:01.217415 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:01.217366    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-39fb96/premium-w6-39fb96-kserve-b9cf7f75c-96blx" podUID="18ca0a94-c322-41fa-9fd9-0f14419b5075" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded"
Oct 30 21:01:01.387018 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:01.386906    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a8f94/premium-w3-5a8f94-kserve-bb96b6848-5hk8h" podUID="24ccc37d-027a-4097-8da5-04df417aef94" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:01:01.447415 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:01.439421    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c4b379/premium-w1-c4b379-kserve-575f896d49-2n6gz" podUID="2aac3e8e-d631-4b9c-bdbb-3b4b1df133a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:01:01.447415 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:01.440626    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-32a969/simulator-w2-32a969-kserve-6cc8dfc44c-dwhlt" podUID="62a6adf7-d17a-4fec-a707-a2d549b67753" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Oct 30 21:01:01.487364 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:01.477605    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-32a969/distinct-2-w2-32a969-kserve-5c7845448b-vt5gw" podUID="d430ed01-65d1-43a6-a0c4-ab82212376a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:01:01.627387 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:01.609695    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a0c5b/premium-w0-4a0c5b-kserve-85dcf89f57-mscdn" podUID="00e42758-77ce-4663-adfa-2202c0aae24a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:01:01.745215 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:01.745178    4521 patch_prober.go:29] interesting pod/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-w5lzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Oct 30 21:01:01.745470 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:01.745444    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-w5lzs" podUID="b7f18233-f19c-4f74-a071-62d63402691c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Oct 30 21:01:01.792343 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:01.788515    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a8f94/distinct-w3-5a8f94-kserve-5f8b544d59-mf95j" podUID="b8ac1cca-4ebd-474c-b32f-a3595a4cd847" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:01:01.792343 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:01.790860    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-32a969/distinct-w2-32a969-kserve-7f798b664f-jdh82" podUID="887f1d27-bf24-4b57-b6e4-8fb095df7a50" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:01:01.811902 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:01.811861    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-32a969/distinct-w2-32a969-kserve-7f798b664f-jdh82"
Oct 30 21:01:01.837728 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:01.837684    4521 patch_prober.go:29] interesting pod/e2e-worker-w0-4a0c5b-openshift-default-676865df99-jjmw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 30 21:01:01.837909 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:01.837757    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-jjmw4" podUID="d91a4f29-950e-4370-92e0-8ec493b72620" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 30 21:01:01.842470 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:01.842429    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c2d1eb/premium-w4-c2d1eb-kserve-8696fbd9b9-mxqhm" podUID="4bb0f898-b77f-419d-bff4-6413e82ef51d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:01:02.029574 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:02.029478    4521 patch_prober.go:29] interesting pod/e2e-worker-w2-32a969-openshift-default-8945d87fc-p55rm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Oct 30 21:01:02.030482 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:02.030448    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-p55rm" podUID="baad1994-acb5-4aae-bb65-a5e8a5b8afdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Oct 30 21:01:02.030932 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:02.030132    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5a8f94/distinct-w3-5a8f94-kserve-5f8b544d59-mf95j"
Oct 30 21:01:02.060390 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:02.029478    4521 patch_prober.go:29] interesting pod/e2e-worker-w2-32a969-openshift-default-8945d87fc-hmwtj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 30 21:01:02.060677 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:02.060643    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-hmwtj" podUID="f6f47707-ed2c-4860-a13a-b1a5d9d1bbe3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 30 21:01:02.077368 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:02.070913    4521 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-5a8f94/premium-w3-5a8f94-kserve-bb96b6848-5hk8h" podUID="24ccc37d-027a-4097-8da5-04df417aef94" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:01:02.155313 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:02.113337    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a8f94/simulator-w3-5a8f94-kserve-77cc88cc89-n4sl6" podUID="1fa6900c-eee2-4f00-b0ec-248def48dfc4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:01:02.155313 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:02.115340    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-65968b/premium-w5-65968b-kserve-67977485f7-c9jpv" podUID="92b2dd65-8a83-436d-afa8-50e5f0d11fa0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:01:02.310646 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:02.310543    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-39fb96/premium-w6-39fb96-kserve-b9cf7f75c-96blx" podUID="18ca0a94-c322-41fa-9fd9-0f14419b5075" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:01:02.354537 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:02.329444    4521 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-c4b379/premium-w1-c4b379-kserve-575f896d49-2n6gz" podUID="2aac3e8e-d631-4b9c-bdbb-3b4b1df133a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": context deadline exceeded"
Oct 30 21:01:02.421505 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:02.421454    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a8f94/premium-w3-5a8f94-kserve-bb96b6848-5hk8h" podUID="24ccc37d-027a-4097-8da5-04df417aef94" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:01:02.445932 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:02.445885    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-32a969/simulator-w2-32a969-kserve-6cc8dfc44c-dwhlt" podUID="62a6adf7-d17a-4fec-a707-a2d549b67753" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Oct 30 21:01:02.446611 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:02.446581    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c4b379/premium-w1-c4b379-kserve-575f896d49-2n6gz" podUID="2aac3e8e-d631-4b9c-bdbb-3b4b1df133a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 30 21:01:02.447142 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:02.447120    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c4b379/premium-w1-c4b379-kserve-575f896d49-2n6gz"
Oct 30 21:01:02.487906 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:02.487849    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-32a969/distinct-2-w2-32a969-kserve-5c7845448b-vt5gw" podUID="d430ed01-65d1-43a6-a0c4-ab82212376a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:01:02.775715 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:02.763663    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a0c5b/premium-w0-4a0c5b-kserve-85dcf89f57-mscdn" podUID="00e42758-77ce-4663-adfa-2202c0aae24a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:01:02.796055 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:02.794431    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-32a969/distinct-w2-32a969-kserve-7f798b664f-jdh82" podUID="887f1d27-bf24-4b57-b6e4-8fb095df7a50" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:01:02.862083 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:02.861457    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c2d1eb/premium-w4-c2d1eb-kserve-8696fbd9b9-mxqhm" podUID="4bb0f898-b77f-419d-bff4-6413e82ef51d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:01:02.862083 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:02.861677    4521 patch_prober.go:29] interesting pod/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-w5lzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Oct 30 21:01:02.862083 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:02.861723    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-w5lzs" podUID="b7f18233-f19c-4f74-a071-62d63402691c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Oct 30 21:01:02.862083 ip-10-0-131-125 kubenswrapper[4521]: W1030 21:01:02.861808    4521 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod655b039a_00a8_40fd_baaa_e58a80149a06.slice/crio-de66fea4c121b98b3749e03500c66fdebcf9ce701a64758ecb3e3627f7e03856 WatchSource:0}: Error finding container de66fea4c121b98b3749e03500c66fdebcf9ce701a64758ecb3e3627f7e03856: Status 404 returned error can't find the container with id de66fea4c121b98b3749e03500c66fdebcf9ce701a64758ecb3e3627f7e03856
Oct 30 21:01:02.904057 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:02.869486    4521 patch_prober.go:29] interesting pod/e2e-worker-w0-4a0c5b-openshift-default-676865df99-jjmw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 30 21:01:02.904057 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:02.869548    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-jjmw4" podUID="d91a4f29-950e-4370-92e0-8ec493b72620" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 30 21:01:02.949644 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:02.949610    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c2d1eb/premium-w4-c2d1eb-kserve-8696fbd9b9-mxqhm"
Oct 30 21:01:02.995590 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:02.995548    4521 patch_prober.go:29] interesting pod/e2e-worker-w2-32a969-openshift-default-8945d87fc-p55rm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Oct 30 21:01:03.015544 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:03.015477    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-p55rm" podUID="baad1994-acb5-4aae-bb65-a5e8a5b8afdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Oct 30 21:01:03.017257 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:03.017222    4521 patch_prober.go:29] interesting pod/e2e-worker-w2-32a969-openshift-default-8945d87fc-hmwtj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 30 21:01:03.017492 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:03.017470    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-hmwtj" podUID="f6f47707-ed2c-4860-a13a-b1a5d9d1bbe3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 30 21:01:03.047568 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:03.047525    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a8f94/distinct-w3-5a8f94-kserve-5f8b544d59-mf95j" podUID="b8ac1cca-4ebd-474c-b32f-a3595a4cd847" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:01:03.135464 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:03.135409    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-847c3a60-bzqgs"]
Oct 30 21:01:03.185407 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:03.163649    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-65968b/premium-w5-65968b-kserve-67977485f7-c9jpv" podUID="92b2dd65-8a83-436d-afa8-50e5f0d11fa0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:01:03.200489 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:03.200392    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-65968b/premium-w5-65968b-kserve-67977485f7-c9jpv"
Oct 30 21:01:03.409344 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:03.408683    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-847c3a60-bzqgs" event={"ID":"655b039a-00a8-40fd-baaa-e58a80149a06","Type":"ContainerStarted","Data":"de66fea4c121b98b3749e03500c66fdebcf9ce701a64758ecb3e3627f7e03856"}
Oct 30 21:01:03.412074 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:03.411426    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a8f94/simulator-w3-5a8f94-kserve-77cc88cc89-n4sl6" podUID="1fa6900c-eee2-4f00-b0ec-248def48dfc4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:01:03.478588 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:03.412300    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-39fb96/premium-w6-39fb96-kserve-b9cf7f75c-96blx" podUID="18ca0a94-c322-41fa-9fd9-0f14419b5075" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:01:03.478588 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:03.423651    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5a8f94/premium-w3-5a8f94-kserve-bb96b6848-5hk8h"
Oct 30 21:01:03.478588 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:03.453110    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c4b379/premium-w1-c4b379-kserve-575f896d49-2n6gz"
Oct 30 21:01:03.478588 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:03.455521    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-32a969/simulator-w2-32a969-kserve-6cc8dfc44c-dwhlt" podUID="62a6adf7-d17a-4fec-a707-a2d549b67753" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:01:03.547508 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:03.547465    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-32a969/distinct-2-w2-32a969-kserve-5c7845448b-vt5gw" podUID="d430ed01-65d1-43a6-a0c4-ab82212376a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:01:03.720356 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:03.718562    4521 patch_prober.go:29] interesting pod/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-w5lzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Oct 30 21:01:03.720639 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:03.720606    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-w5lzs" podUID="b7f18233-f19c-4f74-a071-62d63402691c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Oct 30 21:01:03.790404 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:03.766742    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a0c5b/premium-w0-4a0c5b-kserve-85dcf89f57-mscdn" podUID="00e42758-77ce-4663-adfa-2202c0aae24a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:01:03.814522 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:03.813857    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-39fb96/premium-w6-39fb96-kserve-b9cf7f75c-96blx"
Oct 30 21:01:03.834349 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:03.815395    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-32a969/distinct-w2-32a969-kserve-7f798b664f-jdh82" podUID="887f1d27-bf24-4b57-b6e4-8fb095df7a50" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded"
Oct 30 21:01:03.854357 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:03.835058    4521 patch_prober.go:29] interesting pod/e2e-worker-w0-4a0c5b-openshift-default-676865df99-jjmw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 30 21:01:03.854667 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:03.854628    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-jjmw4" podUID="d91a4f29-950e-4370-92e0-8ec493b72620" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 30 21:01:03.854814 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:03.844215    4521 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-5a8f94/distinct-2-w3-5a8f94-kserve-ffcc6449b-hv48k"
Oct 30 21:01:03.912344 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:03.911454    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c2d1eb/premium-w4-c2d1eb-kserve-8696fbd9b9-mxqhm" podUID="4bb0f898-b77f-419d-bff4-6413e82ef51d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:01:03.984767 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:03.976643    4521 patch_prober.go:29] interesting pod/e2e-worker-w2-32a969-openshift-default-8945d87fc-hmwtj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 30 21:01:03.984767 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:03.982396    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-hmwtj" podUID="f6f47707-ed2c-4860-a13a-b1a5d9d1bbe3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 30 21:01:04.060434 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:04.058760    4521 patch_prober.go:29] interesting pod/e2e-worker-w2-32a969-openshift-default-8945d87fc-p55rm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Oct 30 21:01:04.060434 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:04.058823    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-p55rm" podUID="baad1994-acb5-4aae-bb65-a5e8a5b8afdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Oct 30 21:01:04.494382 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:04.427654    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a8f94/simulator-w3-5a8f94-kserve-77cc88cc89-n4sl6" podUID="1fa6900c-eee2-4f00-b0ec-248def48dfc4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:01:04.494382 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:04.428555    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5a8f94/simulator-w3-5a8f94-kserve-77cc88cc89-n4sl6"
Oct 30 21:01:04.494382 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:04.459709    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c4b379/premium-w1-c4b379-kserve-575f896d49-2n6gz" podUID="2aac3e8e-d631-4b9c-bdbb-3b4b1df133a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 30 21:01:04.525245 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:04.524449    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-32a969/simulator-w2-32a969-kserve-6cc8dfc44c-dwhlt" podUID="62a6adf7-d17a-4fec-a707-a2d549b67753" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:01:04.553379 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:04.551916    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-32a969/distinct-2-w2-32a969-kserve-5c7845448b-vt5gw" podUID="d430ed01-65d1-43a6-a0c4-ab82212376a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:01:04.762897 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:04.756065    4521 patch_prober.go:29] interesting pod/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-w5lzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Oct 30 21:01:04.762897 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:04.756127    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-w5lzs" podUID="b7f18233-f19c-4f74-a071-62d63402691c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Oct 30 21:01:04.837346 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:04.815165    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a0c5b/premium-w0-4a0c5b-kserve-85dcf89f57-mscdn" podUID="00e42758-77ce-4663-adfa-2202c0aae24a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:01:04.840679 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:04.823467    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-32a969/distinct-w2-32a969-kserve-7f798b664f-jdh82" podUID="887f1d27-bf24-4b57-b6e4-8fb095df7a50" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:01:04.961813 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:04.838284    4521 patch_prober.go:29] interesting pod/e2e-worker-w0-4a0c5b-openshift-default-676865df99-jjmw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 30 21:01:05.044350 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:05.019473    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-jjmw4" podUID="d91a4f29-950e-4370-92e0-8ec493b72620" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 30 21:01:05.045670 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:05.003836    4521 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-models-e2e-worker-w3-5a8f94/distinct-2-w3-5a8f94-kserve-ffcc6449b-hv48k" podUID="bc0f5352-5c44-4c35-adf5-3fffab06b644" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/health\": context deadline exceeded"
Oct 30 21:01:05.045914 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:04.988996    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c2d1eb/premium-w4-c2d1eb-kserve-8696fbd9b9-mxqhm" podUID="4bb0f898-b77f-419d-bff4-6413e82ef51d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:01:05.074349 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:05.056343    4521 patch_prober.go:29] interesting pod/e2e-worker-w2-32a969-openshift-default-8945d87fc-p55rm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Oct 30 21:01:05.074349 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:05.056415    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-p55rm" podUID="baad1994-acb5-4aae-bb65-a5e8a5b8afdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Oct 30 21:01:05.074349 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:05.057070    4521 patch_prober.go:29] interesting pod/e2e-worker-w2-32a969-openshift-default-8945d87fc-hmwtj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 30 21:01:05.074349 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:05.057112    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-hmwtj" podUID="f6f47707-ed2c-4860-a13a-b1a5d9d1bbe3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 30 21:01:05.074349 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:05.064309    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a8f94/distinct-w3-5a8f94-kserve-5f8b544d59-mf95j" podUID="b8ac1cca-4ebd-474c-b32f-a3595a4cd847" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded"
Oct 30 21:01:05.120446 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:05.120398    4521 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-5a8f94/distinct-w3-5a8f94-kserve-5f8b544d59-mf95j" podUID="b8ac1cca-4ebd-474c-b32f-a3595a4cd847" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:01:05.201802 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:05.072485    4521 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-32a969/simulator-w2-32a969-kserve-6cc8dfc44c-dwhlt" podUID="62a6adf7-d17a-4fec-a707-a2d549b67753" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:01:05.245347 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:05.230011    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-65968b/premium-w5-65968b-kserve-67977485f7-c9jpv" podUID="92b2dd65-8a83-436d-afa8-50e5f0d11fa0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:01:05.437807 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:05.437760    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a8f94/premium-w3-5a8f94-kserve-bb96b6848-5hk8h" podUID="24ccc37d-027a-4097-8da5-04df417aef94" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:01:05.441048 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:05.441008    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a8f94/simulator-w3-5a8f94-kserve-77cc88cc89-n4sl6" podUID="1fa6900c-eee2-4f00-b0ec-248def48dfc4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 30 21:01:05.465841 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:05.465796    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c4b379/premium-w1-c4b379-kserve-575f896d49-2n6gz" podUID="2aac3e8e-d631-4b9c-bdbb-3b4b1df133a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 30 21:01:05.466801 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:05.466773    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c4b379/premium-w1-c4b379-kserve-575f896d49-2n6gz"
Oct 30 21:01:05.634408 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:05.619400    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-32a969/distinct-2-w2-32a969-kserve-5c7845448b-vt5gw" podUID="d430ed01-65d1-43a6-a0c4-ab82212376a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:01:05.659348 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:05.644762    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-32a969/simulator-w2-32a969-kserve-6cc8dfc44c-dwhlt" podUID="62a6adf7-d17a-4fec-a707-a2d549b67753" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Oct 30 21:01:05.708924 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:05.708892    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c4b379/premium-w1-c4b379-kserve-575f896d49-2n6gz"
Oct 30 21:01:05.795357 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:05.783317    4521 patch_prober.go:29] interesting pod/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-w5lzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Oct 30 21:01:05.795357 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:05.783408    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-w5lzs" podUID="b7f18233-f19c-4f74-a071-62d63402691c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Oct 30 21:01:05.908409 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:05.869958    4521 patch_prober.go:29] interesting pod/e2e-worker-w0-4a0c5b-openshift-default-676865df99-jjmw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 30 21:01:05.908409 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:05.870113    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-jjmw4" podUID="d91a4f29-950e-4370-92e0-8ec493b72620" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 30 21:01:05.966441 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:05.960650    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-32a969/distinct-w2-32a969-kserve-7f798b664f-jdh82"
Oct 30 21:01:05.986569 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:05.986510    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a0c5b/premium-w0-4a0c5b-kserve-85dcf89f57-mscdn" podUID="00e42758-77ce-4663-adfa-2202c0aae24a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:01:06.003453 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:06.003420    4521 patch_prober.go:29] interesting pod/e2e-worker-w2-32a969-openshift-default-8945d87fc-p55rm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Oct 30 21:01:06.031891 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:06.031855    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-p55rm" podUID="baad1994-acb5-4aae-bb65-a5e8a5b8afdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Oct 30 21:01:06.033127 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:06.014184    4521 patch_prober.go:29] interesting pod/e2e-worker-w2-32a969-openshift-default-8945d87fc-hmwtj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 30 21:01:06.033293 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:06.033272    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-hmwtj" podUID="f6f47707-ed2c-4860-a13a-b1a5d9d1bbe3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 30 21:01:06.064553 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:06.055434    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c2d1eb/premium-w4-c2d1eb-kserve-8696fbd9b9-mxqhm" podUID="4bb0f898-b77f-419d-bff4-6413e82ef51d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded"
Oct 30 21:01:06.087197 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:06.070018    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-847c3a60-bzqgs" event={"ID":"655b039a-00a8-40fd-baaa-e58a80149a06","Type":"ContainerStarted","Data":"9d7eaffb7e30a7ea1b321a3b58afe2ef35fcb06713518b8492d9d1336fe1ced9"}
Oct 30 21:01:06.159898 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:06.159691    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-32a969/distinct-2-w2-32a969-kserve-5c7845448b-vt5gw"
Oct 30 21:01:06.182108 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:06.182080    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-32a969/simulator-w2-32a969-kserve-6cc8dfc44c-dwhlt"
Oct 30 21:01:06.202125 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:06.199388    4521 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-847c3a60-bzqgs" podStartSLOduration=7.199370595 podStartE2EDuration="7.199370595s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:00:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:01:06.155547122 +0000 UTC m=+1333.501168410" watchObservedRunningTime="2026-10-30 21:01:06.199370595 +0000 UTC m=+1333.544991835"
Oct 30 21:01:06.207229 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:06.207205    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c2d1eb/premium-w4-c2d1eb-kserve-8696fbd9b9-mxqhm"
Oct 30 21:01:06.253357 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:06.248408    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-65968b/premium-w5-65968b-kserve-67977485f7-c9jpv" podUID="92b2dd65-8a83-436d-afa8-50e5f0d11fa0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:01:06.269375 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:06.262836    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-65968b/premium-w5-65968b-kserve-67977485f7-c9jpv"
Oct 30 21:01:06.269375 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:06.263888    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4a0c5b/premium-w0-4a0c5b-kserve-85dcf89f57-mscdn"
Oct 30 21:01:06.382519 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:06.374448    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-65968b/premium-w5-65968b-kserve-67977485f7-c9jpv"
Oct 30 21:01:06.393250 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:06.392185    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5a8f94/simulator-w3-5a8f94-kserve-77cc88cc89-n4sl6"
Oct 30 21:01:06.696360 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:06.696306    4521 patch_prober.go:29] interesting pod/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-w5lzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Oct 30 21:01:06.696648 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:06.696618    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-w5lzs" podUID="b7f18233-f19c-4f74-a071-62d63402691c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Oct 30 21:01:06.826883 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:06.826839    4521 patch_prober.go:29] interesting pod/e2e-worker-w0-4a0c5b-openshift-default-676865df99-jjmw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 30 21:01:06.827169 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:06.827122    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-jjmw4" podUID="d91a4f29-950e-4370-92e0-8ec493b72620" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 30 21:01:06.987638 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:06.987532    4521 patch_prober.go:29] interesting pod/e2e-worker-w2-32a969-openshift-default-8945d87fc-p55rm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Oct 30 21:01:06.987638 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:06.987615    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-p55rm" podUID="baad1994-acb5-4aae-bb65-a5e8a5b8afdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Oct 30 21:01:07.105414 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:07.104708    4521 generic.go:356] "Generic (PLEG): container finished" podID="655b039a-00a8-40fd-baaa-e58a80149a06" containerID="9d7eaffb7e30a7ea1b321a3b58afe2ef35fcb06713518b8492d9d1336fe1ced9" exitCode=0
Oct 30 21:01:07.106037 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:07.105964    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-847c3a60-bzqgs" event={"ID":"655b039a-00a8-40fd-baaa-e58a80149a06","Type":"ContainerDied","Data":"9d7eaffb7e30a7ea1b321a3b58afe2ef35fcb06713518b8492d9d1336fe1ced9"}
Oct 30 21:01:07.571384 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:07.570901    4521 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-hmwtj"
Oct 30 21:01:07.572842 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:07.572677    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-hmwtj"
Oct 30 21:01:07.574841 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:07.574809    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-hmwtj"
Oct 30 21:01:07.836080 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:07.836006    4521 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-jjmw4"
Oct 30 21:01:07.837248 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:07.836826    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-jjmw4"
Oct 30 21:01:07.838388 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:07.838367    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-jjmw4"
Oct 30 21:01:07.863777 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:07.863744    4521 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-w5lzs"
Oct 30 21:01:07.865052 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:07.864716    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-w5lzs"
Oct 30 21:01:07.865220 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:07.865197    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-w5lzs"
Oct 30 21:01:08.197874 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:08.197841    4521 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-p55rm"
Oct 30 21:01:08.198902 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:08.198876    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-p55rm"
Oct 30 21:01:08.200200 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:08.200179    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-p55rm"
Oct 30 21:01:08.331482 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:08.331455    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-847c3a60-bzqgs"
Oct 30 21:01:08.411366 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:08.411247    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/655b039a-00a8-40fd-baaa-e58a80149a06-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/655b039a-00a8-40fd-baaa-e58a80149a06-maas-api-xks-ca\") pod \"655b039a-00a8-40fd-baaa-e58a80149a06\" (UID: \"655b039a-00a8-40fd-baaa-e58a80149a06\") "
Oct 30 21:01:08.411366 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:08.411339    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/655b039a-00a8-40fd-baaa-e58a80149a06-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/655b039a-00a8-40fd-baaa-e58a80149a06-maas-api-openshift-ca\") pod \"655b039a-00a8-40fd-baaa-e58a80149a06\" (UID: \"655b039a-00a8-40fd-baaa-e58a80149a06\") "
Oct 30 21:01:08.411610 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:08.411451    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/655b039a-00a8-40fd-baaa-e58a80149a06-kube-api-access-wzj5q\" (UniqueName: \"kubernetes.io/projected/655b039a-00a8-40fd-baaa-e58a80149a06-kube-api-access-wzj5q\") pod \"655b039a-00a8-40fd-baaa-e58a80149a06\" (UID: \"655b039a-00a8-40fd-baaa-e58a80149a06\") "
Oct 30 21:01:08.411773 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:08.411736    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/655b039a-00a8-40fd-baaa-e58a80149a06-maas-api-openshift-ca" pod "655b039a-00a8-40fd-baaa-e58a80149a06" (UID: "655b039a-00a8-40fd-baaa-e58a80149a06"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:01:08.412215 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:08.412188    4521 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/655b039a-00a8-40fd-baaa-e58a80149a06-maas-api-openshift-ca\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:01:08.414263 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:08.414236    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/655b039a-00a8-40fd-baaa-e58a80149a06-kube-api-access-wzj5q" pod "655b039a-00a8-40fd-baaa-e58a80149a06" (UID: "655b039a-00a8-40fd-baaa-e58a80149a06"). InnerVolumeSpecName "kube-api-access-wzj5q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:01:08.414506 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:08.414482    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/655b039a-00a8-40fd-baaa-e58a80149a06-maas-api-xks-ca" pod "655b039a-00a8-40fd-baaa-e58a80149a06" (UID: "655b039a-00a8-40fd-baaa-e58a80149a06"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:01:08.513626 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:08.513596    4521 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzj5q\" (UniqueName: \"kubernetes.io/projected/655b039a-00a8-40fd-baaa-e58a80149a06-kube-api-access-wzj5q\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:01:08.513626 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:08.513624    4521 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/655b039a-00a8-40fd-baaa-e58a80149a06-maas-api-xks-ca\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:01:09.121971 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:09.121881    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-847c3a60-bzqgs"
Oct 30 21:01:09.122132 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:09.121890    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-847c3a60-bzqgs" event={"ID":"655b039a-00a8-40fd-baaa-e58a80149a06","Type":"ContainerDied","Data":"de66fea4c121b98b3749e03500c66fdebcf9ce701a64758ecb3e3627f7e03856"}
Oct 30 21:01:09.122132 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:09.122011    4521 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="de66fea4c121b98b3749e03500c66fdebcf9ce701a64758ecb3e3627f7e03856"
Oct 30 21:01:10.948580 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:10.948540    4521 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-r586d"]
Oct 30 21:01:10.949479 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:10.949456    4521 cpu_manager.go:414] "containerMap: removing container" podUID="655b039a-00a8-40fd-baaa-e58a80149a06" containerName="revoke-keys"
Oct 30 21:01:10.949479 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:10.949480    4521 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="655b039a-00a8-40fd-baaa-e58a80149a06" containerName="revoke-keys"
Oct 30 21:01:10.949658 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:10.949620    4521 memory_manager.go:365] "RemoveStaleState removing state" podUID="655b039a-00a8-40fd-baaa-e58a80149a06" containerName="revoke-keys"
Oct 30 21:01:10.953946 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:10.953924    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-r586d"
Oct 30 21:01:10.970341 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:10.969313    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-r586d"]
Oct 30 21:01:11.061635 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:11.061596    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69a73fe6-ec07-446c-aecc-e11806b5f492-istio-envoy\") pod \"e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-r586d\" (UID: \"69a73fe6-ec07-446c-aecc-e11806b5f492\") " pod="openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-r586d"
Oct 30 21:01:11.061830 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:11.061655    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69a73fe6-ec07-446c-aecc-e11806b5f492-istio-podinfo\") pod \"e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-r586d\" (UID: \"69a73fe6-ec07-446c-aecc-e11806b5f492\") " pod="openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-r586d"
Oct 30 21:01:11.061830 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:11.061709    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69a73fe6-ec07-446c-aecc-e11806b5f492-istiod-ca-cert\") pod \"e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-r586d\" (UID: \"69a73fe6-ec07-446c-aecc-e11806b5f492\") " pod="openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-r586d"
Oct 30 21:01:11.061830 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:11.061817    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69a73fe6-ec07-446c-aecc-e11806b5f492-workload-certs\") pod \"e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-r586d\" (UID: \"69a73fe6-ec07-446c-aecc-e11806b5f492\") " pod="openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-r586d"
Oct 30 21:01:11.062026 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:11.061961    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69a73fe6-ec07-446c-aecc-e11806b5f492-credential-socket\") pod \"e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-r586d\" (UID: \"69a73fe6-ec07-446c-aecc-e11806b5f492\") " pod="openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-r586d"
Oct 30 21:01:11.062071 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:11.062027    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69a73fe6-ec07-446c-aecc-e11806b5f492-workload-socket\") pod \"e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-r586d\" (UID: \"69a73fe6-ec07-446c-aecc-e11806b5f492\") " pod="openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-r586d"
Oct 30 21:01:11.062129 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:11.062097    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69a73fe6-ec07-446c-aecc-e11806b5f492-istio-token\") pod \"e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-r586d\" (UID: \"69a73fe6-ec07-446c-aecc-e11806b5f492\") " pod="openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-r586d"
Oct 30 21:01:11.062219 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:11.062198    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4h24\" (UniqueName: \"kubernetes.io/projected/69a73fe6-ec07-446c-aecc-e11806b5f492-kube-api-access-n4h24\") pod \"e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-r586d\" (UID: \"69a73fe6-ec07-446c-aecc-e11806b5f492\") " pod="openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-r586d"
Oct 30 21:01:11.062277 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:11.062239    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69a73fe6-ec07-446c-aecc-e11806b5f492-istio-data\") pod \"e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-r586d\" (UID: \"69a73fe6-ec07-446c-aecc-e11806b5f492\") " pod="openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-r586d"
Oct 30 21:01:11.095254 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:11.095223    4521 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-hz8mp"]
Oct 30 21:01:11.099942 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:11.099794    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-hz8mp"
Oct 30 21:01:11.119729 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:11.119378    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-hz8mp"]
Oct 30 21:01:11.163633 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:11.163602    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69a73fe6-ec07-446c-aecc-e11806b5f492-istio-envoy\") pod \"e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-r586d\" (UID: \"69a73fe6-ec07-446c-aecc-e11806b5f492\") " pod="openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-r586d"
Oct 30 21:01:11.163836 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:11.163645    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69a73fe6-ec07-446c-aecc-e11806b5f492-istio-podinfo\") pod \"e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-r586d\" (UID: \"69a73fe6-ec07-446c-aecc-e11806b5f492\") " pod="openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-r586d"
Oct 30 21:01:11.163836 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:11.163667    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69a73fe6-ec07-446c-aecc-e11806b5f492-istiod-ca-cert\") pod \"e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-r586d\" (UID: \"69a73fe6-ec07-446c-aecc-e11806b5f492\") " pod="openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-r586d"
Oct 30 21:01:11.163836 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:11.163688    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69a73fe6-ec07-446c-aecc-e11806b5f492-workload-certs\") pod \"e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-r586d\" (UID: \"69a73fe6-ec07-446c-aecc-e11806b5f492\") " pod="openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-r586d"
Oct 30 21:01:11.163836 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:11.163741    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69a73fe6-ec07-446c-aecc-e11806b5f492-credential-socket\") pod \"e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-r586d\" (UID: \"69a73fe6-ec07-446c-aecc-e11806b5f492\") " pod="openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-r586d"
Oct 30 21:01:11.163836 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:11.163761    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69a73fe6-ec07-446c-aecc-e11806b5f492-workload-socket\") pod \"e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-r586d\" (UID: \"69a73fe6-ec07-446c-aecc-e11806b5f492\") " pod="openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-r586d"
Oct 30 21:01:11.163836 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:11.163802    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69a73fe6-ec07-446c-aecc-e11806b5f492-istio-token\") pod \"e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-r586d\" (UID: \"69a73fe6-ec07-446c-aecc-e11806b5f492\") " pod="openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-r586d"
Oct 30 21:01:11.163836 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:11.163822    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4h24\" (UniqueName: \"kubernetes.io/projected/69a73fe6-ec07-446c-aecc-e11806b5f492-kube-api-access-n4h24\") pod \"e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-r586d\" (UID: \"69a73fe6-ec07-446c-aecc-e11806b5f492\") " pod="openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-r586d"
Oct 30 21:01:11.163836 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:11.163841    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69a73fe6-ec07-446c-aecc-e11806b5f492-istio-data\") pod \"e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-r586d\" (UID: \"69a73fe6-ec07-446c-aecc-e11806b5f492\") " pod="openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-r586d"
Oct 30 21:01:11.164249 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:11.164192    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69a73fe6-ec07-446c-aecc-e11806b5f492-workload-certs\") pod \"e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-r586d\" (UID: \"69a73fe6-ec07-446c-aecc-e11806b5f492\") " pod="openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-r586d"
Oct 30 21:01:11.164249 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:11.164227    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69a73fe6-ec07-446c-aecc-e11806b5f492-istio-data\") pod \"e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-r586d\" (UID: \"69a73fe6-ec07-446c-aecc-e11806b5f492\") " pod="openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-r586d"
Oct 30 21:01:11.164394 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:11.164338    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69a73fe6-ec07-446c-aecc-e11806b5f492-workload-socket\") pod \"e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-r586d\" (UID: \"69a73fe6-ec07-446c-aecc-e11806b5f492\") " pod="openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-r586d"
Oct 30 21:01:11.164562 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:11.164544    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69a73fe6-ec07-446c-aecc-e11806b5f492-credential-socket\") pod \"e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-r586d\" (UID: \"69a73fe6-ec07-446c-aecc-e11806b5f492\") " pod="openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-r586d"
Oct 30 21:01:11.165042 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:11.165018    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69a73fe6-ec07-446c-aecc-e11806b5f492-istiod-ca-cert\") pod \"e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-r586d\" (UID: \"69a73fe6-ec07-446c-aecc-e11806b5f492\") " pod="openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-r586d"
Oct 30 21:01:11.166616 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:11.166592    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69a73fe6-ec07-446c-aecc-e11806b5f492-istio-envoy\") pod \"e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-r586d\" (UID: \"69a73fe6-ec07-446c-aecc-e11806b5f492\") " pod="openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-r586d"
Oct 30 21:01:11.167579 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:11.167553    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69a73fe6-ec07-446c-aecc-e11806b5f492-istio-podinfo\") pod \"e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-r586d\" (UID: \"69a73fe6-ec07-446c-aecc-e11806b5f492\") " pod="openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-r586d"
Oct 30 21:01:11.193194 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:11.193161    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4h24\" (UniqueName: \"kubernetes.io/projected/69a73fe6-ec07-446c-aecc-e11806b5f492-kube-api-access-n4h24\") pod \"e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-r586d\" (UID: \"69a73fe6-ec07-446c-aecc-e11806b5f492\") " pod="openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-r586d"
Oct 30 21:01:11.194965 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:11.194943    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69a73fe6-ec07-446c-aecc-e11806b5f492-istio-token\") pod \"e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-r586d\" (UID: \"69a73fe6-ec07-446c-aecc-e11806b5f492\") " pod="openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-r586d"
Oct 30 21:01:11.265267 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:11.265170    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b77d06a-b7a8-4b2d-a365-a728d4397d8b-istio-token\") pod \"e2e-worker-w1-c4b379-openshift-default-57fdd6d787-hz8mp\" (UID: \"5b77d06a-b7a8-4b2d-a365-a728d4397d8b\") " pod="openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-hz8mp"
Oct 30 21:01:11.265463 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:11.265311    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b77d06a-b7a8-4b2d-a365-a728d4397d8b-istio-data\") pod \"e2e-worker-w1-c4b379-openshift-default-57fdd6d787-hz8mp\" (UID: \"5b77d06a-b7a8-4b2d-a365-a728d4397d8b\") " pod="openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-hz8mp"
Oct 30 21:01:11.265463 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:11.265394    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6bcv\" (UniqueName: \"kubernetes.io/projected/5b77d06a-b7a8-4b2d-a365-a728d4397d8b-kube-api-access-q6bcv\") pod \"e2e-worker-w1-c4b379-openshift-default-57fdd6d787-hz8mp\" (UID: \"5b77d06a-b7a8-4b2d-a365-a728d4397d8b\") " pod="openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-hz8mp"
Oct 30 21:01:11.265564 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:11.265508    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b77d06a-b7a8-4b2d-a365-a728d4397d8b-credential-socket\") pod \"e2e-worker-w1-c4b379-openshift-default-57fdd6d787-hz8mp\" (UID: \"5b77d06a-b7a8-4b2d-a365-a728d4397d8b\") " pod="openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-hz8mp"
Oct 30 21:01:11.265614 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:11.265571    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b77d06a-b7a8-4b2d-a365-a728d4397d8b-workload-certs\") pod \"e2e-worker-w1-c4b379-openshift-default-57fdd6d787-hz8mp\" (UID: \"5b77d06a-b7a8-4b2d-a365-a728d4397d8b\") " pod="openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-hz8mp"
Oct 30 21:01:11.265614 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:11.265595    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b77d06a-b7a8-4b2d-a365-a728d4397d8b-istiod-ca-cert\") pod \"e2e-worker-w1-c4b379-openshift-default-57fdd6d787-hz8mp\" (UID: \"5b77d06a-b7a8-4b2d-a365-a728d4397d8b\") " pod="openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-hz8mp"
Oct 30 21:01:11.265714 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:11.265702    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b77d06a-b7a8-4b2d-a365-a728d4397d8b-workload-socket\") pod \"e2e-worker-w1-c4b379-openshift-default-57fdd6d787-hz8mp\" (UID: \"5b77d06a-b7a8-4b2d-a365-a728d4397d8b\") " pod="openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-hz8mp"
Oct 30 21:01:11.265747 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:11.265724    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b77d06a-b7a8-4b2d-a365-a728d4397d8b-istio-podinfo\") pod \"e2e-worker-w1-c4b379-openshift-default-57fdd6d787-hz8mp\" (UID: \"5b77d06a-b7a8-4b2d-a365-a728d4397d8b\") " pod="openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-hz8mp"
Oct 30 21:01:11.265777 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:11.265765    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b77d06a-b7a8-4b2d-a365-a728d4397d8b-istio-envoy\") pod \"e2e-worker-w1-c4b379-openshift-default-57fdd6d787-hz8mp\" (UID: \"5b77d06a-b7a8-4b2d-a365-a728d4397d8b\") " pod="openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-hz8mp"
Oct 30 21:01:11.274950 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:11.274921    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-r586d"
Oct 30 21:01:11.366934 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:11.366890    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b77d06a-b7a8-4b2d-a365-a728d4397d8b-istio-data\") pod \"e2e-worker-w1-c4b379-openshift-default-57fdd6d787-hz8mp\" (UID: \"5b77d06a-b7a8-4b2d-a365-a728d4397d8b\") " pod="openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-hz8mp"
Oct 30 21:01:11.367083 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:11.366959    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6bcv\" (UniqueName: \"kubernetes.io/projected/5b77d06a-b7a8-4b2d-a365-a728d4397d8b-kube-api-access-q6bcv\") pod \"e2e-worker-w1-c4b379-openshift-default-57fdd6d787-hz8mp\" (UID: \"5b77d06a-b7a8-4b2d-a365-a728d4397d8b\") " pod="openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-hz8mp"
Oct 30 21:01:11.367083 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:11.367007    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b77d06a-b7a8-4b2d-a365-a728d4397d8b-credential-socket\") pod \"e2e-worker-w1-c4b379-openshift-default-57fdd6d787-hz8mp\" (UID: \"5b77d06a-b7a8-4b2d-a365-a728d4397d8b\") " pod="openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-hz8mp"
Oct 30 21:01:11.367203 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:11.367093    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b77d06a-b7a8-4b2d-a365-a728d4397d8b-workload-certs\") pod \"e2e-worker-w1-c4b379-openshift-default-57fdd6d787-hz8mp\" (UID: \"5b77d06a-b7a8-4b2d-a365-a728d4397d8b\") " pod="openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-hz8mp"
Oct 30 21:01:11.367203 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:11.367127    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b77d06a-b7a8-4b2d-a365-a728d4397d8b-istiod-ca-cert\") pod \"e2e-worker-w1-c4b379-openshift-default-57fdd6d787-hz8mp\" (UID: \"5b77d06a-b7a8-4b2d-a365-a728d4397d8b\") " pod="openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-hz8mp"
Oct 30 21:01:11.367311 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:11.367240    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b77d06a-b7a8-4b2d-a365-a728d4397d8b-workload-socket\") pod \"e2e-worker-w1-c4b379-openshift-default-57fdd6d787-hz8mp\" (UID: \"5b77d06a-b7a8-4b2d-a365-a728d4397d8b\") " pod="openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-hz8mp"
Oct 30 21:01:11.367311 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:11.367270    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b77d06a-b7a8-4b2d-a365-a728d4397d8b-istio-podinfo\") pod \"e2e-worker-w1-c4b379-openshift-default-57fdd6d787-hz8mp\" (UID: \"5b77d06a-b7a8-4b2d-a365-a728d4397d8b\") " pod="openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-hz8mp"
Oct 30 21:01:11.367430 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:11.367350    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b77d06a-b7a8-4b2d-a365-a728d4397d8b-istio-envoy\") pod \"e2e-worker-w1-c4b379-openshift-default-57fdd6d787-hz8mp\" (UID: \"5b77d06a-b7a8-4b2d-a365-a728d4397d8b\") " pod="openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-hz8mp"
Oct 30 21:01:11.367430 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:11.367398    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b77d06a-b7a8-4b2d-a365-a728d4397d8b-istio-token\") pod \"e2e-worker-w1-c4b379-openshift-default-57fdd6d787-hz8mp\" (UID: \"5b77d06a-b7a8-4b2d-a365-a728d4397d8b\") " pod="openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-hz8mp"
Oct 30 21:01:11.368552 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:11.368217    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b77d06a-b7a8-4b2d-a365-a728d4397d8b-istio-data\") pod \"e2e-worker-w1-c4b379-openshift-default-57fdd6d787-hz8mp\" (UID: \"5b77d06a-b7a8-4b2d-a365-a728d4397d8b\") " pod="openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-hz8mp"
Oct 30 21:01:11.368804 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:11.368769    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b77d06a-b7a8-4b2d-a365-a728d4397d8b-workload-socket\") pod \"e2e-worker-w1-c4b379-openshift-default-57fdd6d787-hz8mp\" (UID: \"5b77d06a-b7a8-4b2d-a365-a728d4397d8b\") " pod="openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-hz8mp"
Oct 30 21:01:11.369241 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:11.369030    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b77d06a-b7a8-4b2d-a365-a728d4397d8b-istiod-ca-cert\") pod \"e2e-worker-w1-c4b379-openshift-default-57fdd6d787-hz8mp\" (UID: \"5b77d06a-b7a8-4b2d-a365-a728d4397d8b\") " pod="openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-hz8mp"
Oct 30 21:01:11.369337 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:11.369287    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b77d06a-b7a8-4b2d-a365-a728d4397d8b-credential-socket\") pod \"e2e-worker-w1-c4b379-openshift-default-57fdd6d787-hz8mp\" (UID: \"5b77d06a-b7a8-4b2d-a365-a728d4397d8b\") " pod="openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-hz8mp"
Oct 30 21:01:11.369796 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:11.369551    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b77d06a-b7a8-4b2d-a365-a728d4397d8b-workload-certs\") pod \"e2e-worker-w1-c4b379-openshift-default-57fdd6d787-hz8mp\" (UID: \"5b77d06a-b7a8-4b2d-a365-a728d4397d8b\") " pod="openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-hz8mp"
Oct 30 21:01:11.371807 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:11.371767    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b77d06a-b7a8-4b2d-a365-a728d4397d8b-istio-envoy\") pod \"e2e-worker-w1-c4b379-openshift-default-57fdd6d787-hz8mp\" (UID: \"5b77d06a-b7a8-4b2d-a365-a728d4397d8b\") " pod="openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-hz8mp"
Oct 30 21:01:11.372106 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:11.372088    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b77d06a-b7a8-4b2d-a365-a728d4397d8b-istio-podinfo\") pod \"e2e-worker-w1-c4b379-openshift-default-57fdd6d787-hz8mp\" (UID: \"5b77d06a-b7a8-4b2d-a365-a728d4397d8b\") " pod="openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-hz8mp"
Oct 30 21:01:11.383924 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:11.383900    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b77d06a-b7a8-4b2d-a365-a728d4397d8b-istio-token\") pod \"e2e-worker-w1-c4b379-openshift-default-57fdd6d787-hz8mp\" (UID: \"5b77d06a-b7a8-4b2d-a365-a728d4397d8b\") " pod="openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-hz8mp"
Oct 30 21:01:11.384313 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:11.384296    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6bcv\" (UniqueName: \"kubernetes.io/projected/5b77d06a-b7a8-4b2d-a365-a728d4397d8b-kube-api-access-q6bcv\") pod \"e2e-worker-w1-c4b379-openshift-default-57fdd6d787-hz8mp\" (UID: \"5b77d06a-b7a8-4b2d-a365-a728d4397d8b\") " pod="openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-hz8mp"
Oct 30 21:01:11.416821 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:11.416784    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-hz8mp"
Oct 30 21:01:11.579722 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:11.579690    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-r586d"]
Oct 30 21:01:11.580953 ip-10-0-131-125 kubenswrapper[4521]: W1030 21:01:11.580930    4521 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69a73fe6_ec07_446c_aecc_e11806b5f492.slice/crio-db80057a710ee0a6d92e9cff3022f79cfc41fd58188303461a09d3983b8ce3fa WatchSource:0}: Error finding container db80057a710ee0a6d92e9cff3022f79cfc41fd58188303461a09d3983b8ce3fa: Status 404 returned error can't find the container with id db80057a710ee0a6d92e9cff3022f79cfc41fd58188303461a09d3983b8ce3fa
Oct 30 21:01:11.583958 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:11.583924    4521 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:01:11.584038 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:11.583991    4521 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:01:11.584038 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:11.584021    4521 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:01:11.725656 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:11.725371    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-hz8mp"]
Oct 30 21:01:11.729914 ip-10-0-131-125 kubenswrapper[4521]: W1030 21:01:11.728725    4521 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b77d06a_b7a8_4b2d_a365_a728d4397d8b.slice/crio-620ee961a449611a0340f50fcdefce412cd455cf9730a98fbfb899d909c3f16d WatchSource:0}: Error finding container 620ee961a449611a0340f50fcdefce412cd455cf9730a98fbfb899d909c3f16d: Status 404 returned error can't find the container with id 620ee961a449611a0340f50fcdefce412cd455cf9730a98fbfb899d909c3f16d
Oct 30 21:01:11.732220 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:11.732157    4521 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:01:11.732368 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:11.732225    4521 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:01:11.732368 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:11.732255    4521 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:01:11.762343 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:11.760282    4521 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9bbfd6c-flhst"]
Oct 30 21:01:11.801650 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:11.801618    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9bbfd6c-flhst"
Oct 30 21:01:11.808856 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:11.808683    4521 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c45pw\""
Oct 30 21:01:11.808856 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:11.808796    4521 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:01:11.812433 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:11.812404    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9bbfd6c-flhst"]
Oct 30 21:01:11.979659 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:11.979613    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnjxt\" (UniqueName: \"kubernetes.io/projected/84b77cef-b726-42b7-9917-8e3cd108bcef-kube-api-access-vnjxt\") pod \"maas-api-revoke-sub-e9bbfd6c-flhst\" (UID: \"84b77cef-b726-42b7-9917-8e3cd108bcef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9bbfd6c-flhst"
Oct 30 21:01:11.980607 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:11.980504    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84b77cef-b726-42b7-9917-8e3cd108bcef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e9bbfd6c-flhst\" (UID: \"84b77cef-b726-42b7-9917-8e3cd108bcef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9bbfd6c-flhst"
Oct 30 21:01:11.980726 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:11.980657    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84b77cef-b726-42b7-9917-8e3cd108bcef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e9bbfd6c-flhst\" (UID: \"84b77cef-b726-42b7-9917-8e3cd108bcef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9bbfd6c-flhst"
Oct 30 21:01:12.082160 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:12.082122    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnjxt\" (UniqueName: \"kubernetes.io/projected/84b77cef-b726-42b7-9917-8e3cd108bcef-kube-api-access-vnjxt\") pod \"maas-api-revoke-sub-e9bbfd6c-flhst\" (UID: \"84b77cef-b726-42b7-9917-8e3cd108bcef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9bbfd6c-flhst"
Oct 30 21:01:12.082387 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:12.082255    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84b77cef-b726-42b7-9917-8e3cd108bcef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e9bbfd6c-flhst\" (UID: \"84b77cef-b726-42b7-9917-8e3cd108bcef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9bbfd6c-flhst"
Oct 30 21:01:12.082387 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:12.082291    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84b77cef-b726-42b7-9917-8e3cd108bcef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e9bbfd6c-flhst\" (UID: \"84b77cef-b726-42b7-9917-8e3cd108bcef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9bbfd6c-flhst"
Oct 30 21:01:12.083243 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:12.083205    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84b77cef-b726-42b7-9917-8e3cd108bcef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e9bbfd6c-flhst\" (UID: \"84b77cef-b726-42b7-9917-8e3cd108bcef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9bbfd6c-flhst"
Oct 30 21:01:12.084774 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:12.084750    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84b77cef-b726-42b7-9917-8e3cd108bcef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e9bbfd6c-flhst\" (UID: \"84b77cef-b726-42b7-9917-8e3cd108bcef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9bbfd6c-flhst"
Oct 30 21:01:12.094740 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:12.094683    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnjxt\" (UniqueName: \"kubernetes.io/projected/84b77cef-b726-42b7-9917-8e3cd108bcef-kube-api-access-vnjxt\") pod \"maas-api-revoke-sub-e9bbfd6c-flhst\" (UID: \"84b77cef-b726-42b7-9917-8e3cd108bcef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9bbfd6c-flhst"
Oct 30 21:01:12.144346 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:12.136757    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9bbfd6c-flhst"
Oct 30 21:01:12.240348 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:12.224717    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-r586d" event={"ID":"69a73fe6-ec07-446c-aecc-e11806b5f492","Type":"ContainerStarted","Data":"26904e2c90ba23d519dbf5ec4696939b31e4d84322e700151eec2883e6f6a117"}
Oct 30 21:01:12.240348 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:12.224790    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-r586d" event={"ID":"69a73fe6-ec07-446c-aecc-e11806b5f492","Type":"ContainerStarted","Data":"db80057a710ee0a6d92e9cff3022f79cfc41fd58188303461a09d3983b8ce3fa"}
Oct 30 21:01:12.249914 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:12.249870    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-hz8mp" event={"ID":"5b77d06a-b7a8-4b2d-a365-a728d4397d8b","Type":"ContainerStarted","Data":"1fc06542506386cd87be1aa2f62f7c628d987268e35a2f5c5f557c0249431354"}
Oct 30 21:01:12.250112 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:12.249940    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-hz8mp" event={"ID":"5b77d06a-b7a8-4b2d-a365-a728d4397d8b","Type":"ContainerStarted","Data":"620ee961a449611a0340f50fcdefce412cd455cf9730a98fbfb899d909c3f16d"}
Oct 30 21:01:12.284346 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:12.279007    4521 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-r586d"
Oct 30 21:01:12.284346 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:12.279130    4521 patch_prober.go:29] interesting pod/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-r586d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Oct 30 21:01:12.284346 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:12.279173    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-r586d" podUID="69a73fe6-ec07-446c-aecc-e11806b5f492" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Oct 30 21:01:12.291671 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:12.291608    4521 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-r586d" podStartSLOduration=2.291588079 podStartE2EDuration="2.291588079s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:01:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:01:12.26972482 +0000 UTC m=+1339.615346061" watchObservedRunningTime="2026-10-30 21:01:12.291588079 +0000 UTC m=+1339.637209318"
Oct 30 21:01:12.333387 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:12.328897    4521 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-hz8mp" podStartSLOduration=1.328872329 podStartE2EDuration="1.328872329s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:01:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:01:12.297776824 +0000 UTC m=+1339.643398078" watchObservedRunningTime="2026-10-30 21:01:12.328872329 +0000 UTC m=+1339.674493570"
Oct 30 21:01:12.426851 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:12.426733    4521 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-hz8mp"
Oct 30 21:01:12.439354 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:12.432444    4521 patch_prober.go:29] interesting pod/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-hz8mp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Oct 30 21:01:12.439354 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:12.432508    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-hz8mp" podUID="5b77d06a-b7a8-4b2d-a365-a728d4397d8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Oct 30 21:01:12.974055 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:12.974014    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9bbfd6c-flhst"]
Oct 30 21:01:13.117945 ip-10-0-131-125 kubenswrapper[4521]: E1030 21:01:13.107007    4521 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84b77cef_b726_42b7_9917_8e3cd108bcef.slice/crio-bd9244fe3882d3c74a1e44ee00062cc0fb3ce1a9686146616540418a81ce3ea8\": RecentStats: unable to find data in memory cache]"
Oct 30 21:01:13.333409 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:13.316839    4521 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-c4b379/distinct-2-w1-c4b379-kserve-7999b785f5-97fz5"]
Oct 30 21:01:13.333409 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:13.319215    4521 patch_prober.go:29] interesting pod/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-r586d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Oct 30 21:01:13.333409 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:13.319266    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-r586d" podUID="69a73fe6-ec07-446c-aecc-e11806b5f492" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Oct 30 21:01:13.333409 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:13.321977    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-c4b379/distinct-2-w1-c4b379-kserve-7999b785f5-97fz5"]
Oct 30 21:01:13.333409 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:13.322082    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-c4b379/distinct-2-w1-c4b379-kserve-7999b785f5-97fz5"
Oct 30 21:01:13.429300 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:13.429100    4521 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-c4b379\"/\"distinct-2-w1-c4b379-kserve-self-signed-certs\""
Oct 30 21:01:13.430643 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:13.430617    4521 patch_prober.go:29] interesting pod/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-hz8mp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Oct 30 21:01:13.430819 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:13.430792    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-hz8mp" podUID="5b77d06a-b7a8-4b2d-a365-a728d4397d8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Oct 30 21:01:13.498084 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:13.464186    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/080a654d-a5bf-4845-a07f-cc8a7e6d0ace-home\") pod \"distinct-2-w1-c4b379-kserve-7999b785f5-97fz5\" (UID: \"080a654d-a5bf-4845-a07f-cc8a7e6d0ace\") " pod="e2e-models-e2e-worker-w1-c4b379/distinct-2-w1-c4b379-kserve-7999b785f5-97fz5"
Oct 30 21:01:13.498084 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:13.464244    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/080a654d-a5bf-4845-a07f-cc8a7e6d0ace-tls-certs\") pod \"distinct-2-w1-c4b379-kserve-7999b785f5-97fz5\" (UID: \"080a654d-a5bf-4845-a07f-cc8a7e6d0ace\") " pod="e2e-models-e2e-worker-w1-c4b379/distinct-2-w1-c4b379-kserve-7999b785f5-97fz5"
Oct 30 21:01:13.498084 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:13.464301    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/080a654d-a5bf-4845-a07f-cc8a7e6d0ace-dshm\") pod \"distinct-2-w1-c4b379-kserve-7999b785f5-97fz5\" (UID: \"080a654d-a5bf-4845-a07f-cc8a7e6d0ace\") " pod="e2e-models-e2e-worker-w1-c4b379/distinct-2-w1-c4b379-kserve-7999b785f5-97fz5"
Oct 30 21:01:13.498084 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:13.464341    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/080a654d-a5bf-4845-a07f-cc8a7e6d0ace-model-cache\") pod \"distinct-2-w1-c4b379-kserve-7999b785f5-97fz5\" (UID: \"080a654d-a5bf-4845-a07f-cc8a7e6d0ace\") " pod="e2e-models-e2e-worker-w1-c4b379/distinct-2-w1-c4b379-kserve-7999b785f5-97fz5"
Oct 30 21:01:13.498084 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:13.464477    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zw5t2\" (UniqueName: \"kubernetes.io/projected/080a654d-a5bf-4845-a07f-cc8a7e6d0ace-kube-api-access-zw5t2\") pod \"distinct-2-w1-c4b379-kserve-7999b785f5-97fz5\" (UID: \"080a654d-a5bf-4845-a07f-cc8a7e6d0ace\") " pod="e2e-models-e2e-worker-w1-c4b379/distinct-2-w1-c4b379-kserve-7999b785f5-97fz5"
Oct 30 21:01:13.498084 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:13.464538    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/080a654d-a5bf-4845-a07f-cc8a7e6d0ace-tmp-dir\") pod \"distinct-2-w1-c4b379-kserve-7999b785f5-97fz5\" (UID: \"080a654d-a5bf-4845-a07f-cc8a7e6d0ace\") " pod="e2e-models-e2e-worker-w1-c4b379/distinct-2-w1-c4b379-kserve-7999b785f5-97fz5"
Oct 30 21:01:13.535114 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:13.511151    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9bbfd6c-flhst" event={"ID":"84b77cef-b726-42b7-9917-8e3cd108bcef","Type":"ContainerStarted","Data":"bd9244fe3882d3c74a1e44ee00062cc0fb3ce1a9686146616540418a81ce3ea8"}
Oct 30 21:01:13.569283 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:13.569231    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/080a654d-a5bf-4845-a07f-cc8a7e6d0ace-home\") pod \"distinct-2-w1-c4b379-kserve-7999b785f5-97fz5\" (UID: \"080a654d-a5bf-4845-a07f-cc8a7e6d0ace\") " pod="e2e-models-e2e-worker-w1-c4b379/distinct-2-w1-c4b379-kserve-7999b785f5-97fz5"
Oct 30 21:01:13.570089 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:13.570063    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/080a654d-a5bf-4845-a07f-cc8a7e6d0ace-tls-certs\") pod \"distinct-2-w1-c4b379-kserve-7999b785f5-97fz5\" (UID: \"080a654d-a5bf-4845-a07f-cc8a7e6d0ace\") " pod="e2e-models-e2e-worker-w1-c4b379/distinct-2-w1-c4b379-kserve-7999b785f5-97fz5"
Oct 30 21:01:13.570819 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:13.570797    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/080a654d-a5bf-4845-a07f-cc8a7e6d0ace-dshm\") pod \"distinct-2-w1-c4b379-kserve-7999b785f5-97fz5\" (UID: \"080a654d-a5bf-4845-a07f-cc8a7e6d0ace\") " pod="e2e-models-e2e-worker-w1-c4b379/distinct-2-w1-c4b379-kserve-7999b785f5-97fz5"
Oct 30 21:01:13.570988 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:13.570969    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/080a654d-a5bf-4845-a07f-cc8a7e6d0ace-model-cache\") pod \"distinct-2-w1-c4b379-kserve-7999b785f5-97fz5\" (UID: \"080a654d-a5bf-4845-a07f-cc8a7e6d0ace\") " pod="e2e-models-e2e-worker-w1-c4b379/distinct-2-w1-c4b379-kserve-7999b785f5-97fz5"
Oct 30 21:01:13.577634 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:13.577595    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zw5t2\" (UniqueName: \"kubernetes.io/projected/080a654d-a5bf-4845-a07f-cc8a7e6d0ace-kube-api-access-zw5t2\") pod \"distinct-2-w1-c4b379-kserve-7999b785f5-97fz5\" (UID: \"080a654d-a5bf-4845-a07f-cc8a7e6d0ace\") " pod="e2e-models-e2e-worker-w1-c4b379/distinct-2-w1-c4b379-kserve-7999b785f5-97fz5"
Oct 30 21:01:13.587651 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:13.587527    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/080a654d-a5bf-4845-a07f-cc8a7e6d0ace-tmp-dir\") pod \"distinct-2-w1-c4b379-kserve-7999b785f5-97fz5\" (UID: \"080a654d-a5bf-4845-a07f-cc8a7e6d0ace\") " pod="e2e-models-e2e-worker-w1-c4b379/distinct-2-w1-c4b379-kserve-7999b785f5-97fz5"
Oct 30 21:01:13.593411 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:13.593377    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/080a654d-a5bf-4845-a07f-cc8a7e6d0ace-tmp-dir\") pod \"distinct-2-w1-c4b379-kserve-7999b785f5-97fz5\" (UID: \"080a654d-a5bf-4845-a07f-cc8a7e6d0ace\") " pod="e2e-models-e2e-worker-w1-c4b379/distinct-2-w1-c4b379-kserve-7999b785f5-97fz5"
Oct 30 21:01:13.593562 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:13.578134    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/080a654d-a5bf-4845-a07f-cc8a7e6d0ace-model-cache\") pod \"distinct-2-w1-c4b379-kserve-7999b785f5-97fz5\" (UID: \"080a654d-a5bf-4845-a07f-cc8a7e6d0ace\") " pod="e2e-models-e2e-worker-w1-c4b379/distinct-2-w1-c4b379-kserve-7999b785f5-97fz5"
Oct 30 21:01:13.593562 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:13.570253    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/080a654d-a5bf-4845-a07f-cc8a7e6d0ace-home\") pod \"distinct-2-w1-c4b379-kserve-7999b785f5-97fz5\" (UID: \"080a654d-a5bf-4845-a07f-cc8a7e6d0ace\") " pod="e2e-models-e2e-worker-w1-c4b379/distinct-2-w1-c4b379-kserve-7999b785f5-97fz5"
Oct 30 21:01:13.613813 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:13.608204    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zw5t2\" (UniqueName: \"kubernetes.io/projected/080a654d-a5bf-4845-a07f-cc8a7e6d0ace-kube-api-access-zw5t2\") pod \"distinct-2-w1-c4b379-kserve-7999b785f5-97fz5\" (UID: \"080a654d-a5bf-4845-a07f-cc8a7e6d0ace\") " pod="e2e-models-e2e-worker-w1-c4b379/distinct-2-w1-c4b379-kserve-7999b785f5-97fz5"
Oct 30 21:01:13.613813 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:13.609346    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/080a654d-a5bf-4845-a07f-cc8a7e6d0ace-tls-certs\") pod \"distinct-2-w1-c4b379-kserve-7999b785f5-97fz5\" (UID: \"080a654d-a5bf-4845-a07f-cc8a7e6d0ace\") " pod="e2e-models-e2e-worker-w1-c4b379/distinct-2-w1-c4b379-kserve-7999b785f5-97fz5"
Oct 30 21:01:13.680457 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:13.663658    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/080a654d-a5bf-4845-a07f-cc8a7e6d0ace-dshm\") pod \"distinct-2-w1-c4b379-kserve-7999b785f5-97fz5\" (UID: \"080a654d-a5bf-4845-a07f-cc8a7e6d0ace\") " pod="e2e-models-e2e-worker-w1-c4b379/distinct-2-w1-c4b379-kserve-7999b785f5-97fz5"
Oct 30 21:01:13.878063 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:13.876689    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-c4b379/distinct-2-w1-c4b379-kserve-7999b785f5-97fz5"
Oct 30 21:01:13.985237 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:13.985195    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-32a969/distinct-w2-32a969-kserve-7f798b664f-jdh82" podUID="887f1d27-bf24-4b57-b6e4-8fb095df7a50" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:01:14.198348 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:14.188632    4521 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-5a8f94/distinct-2-w3-5a8f94-kserve-ffcc6449b-hv48k"
Oct 30 21:01:14.243349 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:14.232906    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5a8f94/distinct-2-w3-5a8f94-kserve-ffcc6449b-hv48k"
Oct 30 21:01:14.281097 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:14.279942    4521 patch_prober.go:29] interesting pod/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-r586d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Oct 30 21:01:14.281097 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:14.280016    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-r586d" podUID="69a73fe6-ec07-446c-aecc-e11806b5f492" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Oct 30 21:01:14.377347 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:14.376769    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-39fb96/premium-w6-39fb96-kserve-b9cf7f75c-96blx" podUID="18ca0a94-c322-41fa-9fd9-0f14419b5075" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded"
Oct 30 21:01:14.477404 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:14.442920    4521 patch_prober.go:29] interesting pod/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-hz8mp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Oct 30 21:01:14.477404 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:14.443020    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-hz8mp" podUID="5b77d06a-b7a8-4b2d-a365-a728d4397d8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Oct 30 21:01:14.519949 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:14.497340    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-32a969/distinct-2-w2-32a969-kserve-5c7845448b-vt5gw" podUID="d430ed01-65d1-43a6-a0c4-ab82212376a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded"
Oct 30 21:01:14.757214 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:14.755267    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9bbfd6c-flhst" event={"ID":"84b77cef-b726-42b7-9917-8e3cd108bcef","Type":"ContainerStarted","Data":"b3bea3711ed721cb76ccfc76f061ab00a36a59930056e05e1f24814db86d448e"}
Oct 30 21:01:14.950431 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:14.948613    4521 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9bbfd6c-flhst" podStartSLOduration=3.948588292 podStartE2EDuration="3.948588292s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:01:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:01:14.84068971 +0000 UTC m=+1342.186310951" watchObservedRunningTime="2026-10-30 21:01:14.948588292 +0000 UTC m=+1342.294209532"
Oct 30 21:01:15.276547 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:15.275768    4521 patch_prober.go:29] interesting pod/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-r586d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Oct 30 21:01:15.276547 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:15.275835    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-r586d" podUID="69a73fe6-ec07-446c-aecc-e11806b5f492" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Oct 30 21:01:15.607290 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:15.607244    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-32a969/distinct-2-w2-32a969-kserve-5c7845448b-vt5gw" podUID="d430ed01-65d1-43a6-a0c4-ab82212376a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:01:15.609393 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:15.608141    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-65968b/premium-w5-65968b-kserve-67977485f7-c9jpv" podUID="92b2dd65-8a83-436d-afa8-50e5f0d11fa0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 30 21:01:15.609393 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:15.608834    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-32a969/distinct-2-w2-32a969-kserve-5c7845448b-vt5gw"
Oct 30 21:01:15.631165 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:15.630487    4521 patch_prober.go:29] interesting pod/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-hz8mp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Oct 30 21:01:15.631165 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:15.630542    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-hz8mp" podUID="5b77d06a-b7a8-4b2d-a365-a728d4397d8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Oct 30 21:01:15.967594 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:15.964510    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-c4b379/distinct-2-w1-c4b379-kserve-7999b785f5-97fz5"]
Oct 30 21:01:15.967594 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:15.964599    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-32a969/distinct-w2-32a969-kserve-7f798b664f-jdh82" podUID="887f1d27-bf24-4b57-b6e4-8fb095df7a50" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:01:16.000214 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:16.000175    4521 generic.go:356] "Generic (PLEG): container finished" podID="84b77cef-b726-42b7-9917-8e3cd108bcef" containerID="b3bea3711ed721cb76ccfc76f061ab00a36a59930056e05e1f24814db86d448e" exitCode=0
Oct 30 21:01:16.000604 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:16.000577    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9bbfd6c-flhst" event={"ID":"84b77cef-b726-42b7-9917-8e3cd108bcef","Type":"ContainerDied","Data":"b3bea3711ed721cb76ccfc76f061ab00a36a59930056e05e1f24814db86d448e"}
Oct 30 21:01:16.061350 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:16.050289    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a8f94/distinct-w3-5a8f94-kserve-5f8b544d59-mf95j" podUID="b8ac1cca-4ebd-474c-b32f-a3595a4cd847" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:01:16.077052 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:16.077017    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-c4b379/distinct-2-w1-c4b379-kserve-7999b785f5-97fz5" event={"ID":"080a654d-a5bf-4845-a07f-cc8a7e6d0ace","Type":"ContainerStarted","Data":"f2e2e02d26328846b91e36984c550935a9a110628cf8a7520351633ac247fcb4"}
Oct 30 21:01:16.256380 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:16.254505    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a8f94/distinct-2-w3-5a8f94-kserve-ffcc6449b-hv48k" podUID="bc0f5352-5c44-4c35-adf5-3fffab06b644" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": context deadline exceeded"
Oct 30 21:01:16.296354 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:16.294861    4521 patch_prober.go:29] interesting pod/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-r586d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Oct 30 21:01:16.296354 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:16.294929    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-r586d" podUID="69a73fe6-ec07-446c-aecc-e11806b5f492" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Oct 30 21:01:16.428416 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:16.425205    4521 patch_prober.go:29] interesting pod/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-hz8mp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Oct 30 21:01:16.453565 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:16.453502    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-hz8mp" podUID="5b77d06a-b7a8-4b2d-a365-a728d4397d8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Oct 30 21:01:16.608635 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:16.608509    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-32a969/distinct-2-w2-32a969-kserve-5c7845448b-vt5gw" podUID="d430ed01-65d1-43a6-a0c4-ab82212376a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded"
Oct 30 21:01:16.631496 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:16.608871    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a8f94/simulator-w3-5a8f94-kserve-77cc88cc89-n4sl6" podUID="1fa6900c-eee2-4f00-b0ec-248def48dfc4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 30 21:01:16.636486 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:16.608917    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-65968b/premium-w5-65968b-kserve-67977485f7-c9jpv" podUID="92b2dd65-8a83-436d-afa8-50e5f0d11fa0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:01:16.636486 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:16.632911    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-65968b/premium-w5-65968b-kserve-67977485f7-c9jpv"
Oct 30 21:01:16.636486 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:16.634713    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-39fb96/premium-w6-39fb96-kserve-b9cf7f75c-96blx" podUID="18ca0a94-c322-41fa-9fd9-0f14419b5075" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded"
Oct 30 21:01:16.639305 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:16.638146    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-32a969/simulator-w2-32a969-kserve-6cc8dfc44c-dwhlt" podUID="62a6adf7-d17a-4fec-a707-a2d549b67753" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Oct 30 21:01:16.639305 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:16.638787    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a8f94/premium-w3-5a8f94-kserve-bb96b6848-5hk8h" podUID="24ccc37d-027a-4097-8da5-04df417aef94" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Oct 30 21:01:16.683646 ip-10-0-131-125 kubenswrapper[4521]: E1030 21:01:16.683508    4521 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lkn2k" containerName="extractor"
Oct 30 21:01:16.823727 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:16.823134    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c4b379/premium-w1-c4b379-kserve-575f896d49-2n6gz" podUID="2aac3e8e-d631-4b9c-bdbb-3b4b1df133a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 30 21:01:16.852393 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:16.833872    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a0c5b/premium-w0-4a0c5b-kserve-85dcf89f57-mscdn" podUID="00e42758-77ce-4663-adfa-2202c0aae24a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Oct 30 21:01:16.968351 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:16.964634    4521 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-c2d1eb/premium-w4-c2d1eb-kserve-8696fbd9b9-mxqhm" podUID="4bb0f898-b77f-419d-bff4-6413e82ef51d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:01:16.968351 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:16.964980    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-32a969/distinct-w2-32a969-kserve-7f798b664f-jdh82" podUID="887f1d27-bf24-4b57-b6e4-8fb095df7a50" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:01:16.968351 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:16.965689    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-32a969/distinct-w2-32a969-kserve-7f798b664f-jdh82"
Oct 30 21:01:17.059858 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:17.059811    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a8f94/distinct-w3-5a8f94-kserve-5f8b544d59-mf95j" podUID="b8ac1cca-4ebd-474c-b32f-a3595a4cd847" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:01:17.060810 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:17.060787    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5a8f94/distinct-w3-5a8f94-kserve-5f8b544d59-mf95j"
Oct 30 21:01:17.076392 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:17.076278    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-32a969/distinct-w2-32a969-kserve-7f798b664f-jdh82"
Oct 30 21:01:17.104293 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:17.102901    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-32a969/distinct-2-w2-32a969-kserve-5c7845448b-vt5gw"
Oct 30 21:01:17.113349 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:17.109958    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5a8f94/distinct-w3-5a8f94-kserve-5f8b544d59-mf95j"
Oct 30 21:01:17.201059 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:17.200451    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-65968b/premium-w5-65968b-kserve-67977485f7-c9jpv"
Oct 30 21:01:17.285779 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:17.285748    4521 patch_prober.go:29] interesting pod/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-r586d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Oct 30 21:01:17.285982 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:17.285961    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-r586d" podUID="69a73fe6-ec07-446c-aecc-e11806b5f492" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Oct 30 21:01:17.444409 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:17.420739    4521 patch_prober.go:29] interesting pod/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-hz8mp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Oct 30 21:01:17.444409 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:17.420801    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-hz8mp" podUID="5b77d06a-b7a8-4b2d-a365-a728d4397d8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Oct 30 21:01:17.705814 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:17.705750    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9bbfd6c-flhst"
Oct 30 21:01:17.785203 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:17.783962    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84b77cef-b726-42b7-9917-8e3cd108bcef-kube-api-access-vnjxt\" (UniqueName: \"kubernetes.io/projected/84b77cef-b726-42b7-9917-8e3cd108bcef-kube-api-access-vnjxt\") pod \"84b77cef-b726-42b7-9917-8e3cd108bcef\" (UID: \"84b77cef-b726-42b7-9917-8e3cd108bcef\") "
Oct 30 21:01:17.785585 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:17.785555    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84b77cef-b726-42b7-9917-8e3cd108bcef-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84b77cef-b726-42b7-9917-8e3cd108bcef-maas-api-openshift-ca\") pod \"84b77cef-b726-42b7-9917-8e3cd108bcef\" (UID: \"84b77cef-b726-42b7-9917-8e3cd108bcef\") "
Oct 30 21:01:17.785750 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:17.785729    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84b77cef-b726-42b7-9917-8e3cd108bcef-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84b77cef-b726-42b7-9917-8e3cd108bcef-maas-api-xks-ca\") pod \"84b77cef-b726-42b7-9917-8e3cd108bcef\" (UID: \"84b77cef-b726-42b7-9917-8e3cd108bcef\") "
Oct 30 21:01:17.786024 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:17.785993    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84b77cef-b726-42b7-9917-8e3cd108bcef-maas-api-openshift-ca" pod "84b77cef-b726-42b7-9917-8e3cd108bcef" (UID: "84b77cef-b726-42b7-9917-8e3cd108bcef"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:01:17.787005 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:17.786962    4521 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84b77cef-b726-42b7-9917-8e3cd108bcef-maas-api-openshift-ca\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:01:17.797252 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:17.790791    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84b77cef-b726-42b7-9917-8e3cd108bcef-kube-api-access-vnjxt" pod "84b77cef-b726-42b7-9917-8e3cd108bcef" (UID: "84b77cef-b726-42b7-9917-8e3cd108bcef"). InnerVolumeSpecName "kube-api-access-vnjxt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:01:17.797563 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:17.797541    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84b77cef-b726-42b7-9917-8e3cd108bcef-maas-api-xks-ca" pod "84b77cef-b726-42b7-9917-8e3cd108bcef" (UID: "84b77cef-b726-42b7-9917-8e3cd108bcef"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:01:17.888825 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:17.888783    4521 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vnjxt\" (UniqueName: \"kubernetes.io/projected/84b77cef-b726-42b7-9917-8e3cd108bcef-kube-api-access-vnjxt\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:01:17.888825 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:17.888817    4521 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84b77cef-b726-42b7-9917-8e3cd108bcef-maas-api-xks-ca\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:01:18.187317 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:18.187283    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-c4b379/distinct-2-w1-c4b379-kserve-7999b785f5-97fz5" event={"ID":"080a654d-a5bf-4845-a07f-cc8a7e6d0ace","Type":"ContainerStarted","Data":"89bfd3d479cd779cb5afde54eac0acbb025cec7a5720c730ff0aa0ea1a55c2f0"}
Oct 30 21:01:18.188986 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:18.188953    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9bbfd6c-flhst" event={"ID":"84b77cef-b726-42b7-9917-8e3cd108bcef","Type":"ContainerDied","Data":"bd9244fe3882d3c74a1e44ee00062cc0fb3ce1a9686146616540418a81ce3ea8"}
Oct 30 21:01:18.188986 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:18.188974    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9bbfd6c-flhst"
Oct 30 21:01:18.189150 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:18.189000    4521 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bd9244fe3882d3c74a1e44ee00062cc0fb3ce1a9686146616540418a81ce3ea8"
Oct 30 21:01:18.250349 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:18.209769    4521 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-c4b379/distinct-2-w1-c4b379-kserve-7999b785f5-97fz5" podStartSLOduration=6.209750566 podStartE2EDuration="6.209750566s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:01:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:01:18.208905601 +0000 UTC m=+1345.554526841" watchObservedRunningTime="2026-10-30 21:01:18.209750566 +0000 UTC m=+1345.555371807"
Oct 30 21:01:18.278378 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:18.275366    4521 patch_prober.go:29] interesting pod/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-r586d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Oct 30 21:01:18.278378 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:18.275425    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-r586d" podUID="69a73fe6-ec07-446c-aecc-e11806b5f492" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Oct 30 21:01:18.429576 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:18.424133    4521 patch_prober.go:29] interesting pod/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-hz8mp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Oct 30 21:01:18.429576 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:18.424313    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-hz8mp" podUID="5b77d06a-b7a8-4b2d-a365-a728d4397d8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Oct 30 21:01:18.882615 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:18.881439    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c4b379/distinct-2-w1-c4b379-kserve-7999b785f5-97fz5"
Oct 30 21:01:19.276026 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:19.275989    4521 patch_prober.go:29] interesting pod/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-r586d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Oct 30 21:01:19.276249 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:19.276050    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-r586d" podUID="69a73fe6-ec07-446c-aecc-e11806b5f492" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Oct 30 21:01:19.427406 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:19.427356    4521 patch_prober.go:29] interesting pod/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-hz8mp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Oct 30 21:01:19.427777 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:19.427652    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-hz8mp" podUID="5b77d06a-b7a8-4b2d-a365-a728d4397d8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Oct 30 21:01:20.275457 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:20.275417    4521 patch_prober.go:29] interesting pod/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-r586d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Oct 30 21:01:20.276004 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:20.275490    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-r586d" podUID="69a73fe6-ec07-446c-aecc-e11806b5f492" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Oct 30 21:01:20.417818 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:20.417779    4521 patch_prober.go:29] interesting pod/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-hz8mp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Oct 30 21:01:20.418088 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:20.418056    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-hz8mp" podUID="5b77d06a-b7a8-4b2d-a365-a728d4397d8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Oct 30 21:01:21.281007 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:21.280976    4521 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-r586d"
Oct 30 21:01:21.281456 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:21.281437    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-r586d"
Oct 30 21:01:21.282233 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:21.282214    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-r586d"
Oct 30 21:01:21.423013 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:21.422983    4521 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-hz8mp"
Oct 30 21:01:21.423388 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:21.423363    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-hz8mp"
Oct 30 21:01:21.424077 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:21.424060    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-hz8mp"
Oct 30 21:01:23.877403 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:23.877367    4521 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-c4b379/distinct-2-w1-c4b379-kserve-7999b785f5-97fz5"
Oct 30 21:01:23.890863 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:23.890829    4521 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-c4b379/distinct-2-w1-c4b379-kserve-7999b785f5-97fz5"
Oct 30 21:01:24.267814 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:24.267785    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c4b379/distinct-2-w1-c4b379-kserve-7999b785f5-97fz5"
Oct 30 21:01:30.139308 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:30.139269    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4kxb"]
Oct 30 21:01:30.388468 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:30.388433    4521 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f768d99c-rp29v"]
Oct 30 21:01:30.389080 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:30.389060    4521 cpu_manager.go:414] "containerMap: removing container" podUID="84b77cef-b726-42b7-9917-8e3cd108bcef" containerName="revoke-keys"
Oct 30 21:01:30.389188 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:30.389082    4521 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84b77cef-b726-42b7-9917-8e3cd108bcef" containerName="revoke-keys"
Oct 30 21:01:30.389254 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:30.389240    4521 memory_manager.go:365] "RemoveStaleState removing state" podUID="84b77cef-b726-42b7-9917-8e3cd108bcef" containerName="revoke-keys"
Oct 30 21:01:30.394391 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:30.394316    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f768d99c-rp29v"
Oct 30 21:01:30.399577 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:30.399549    4521 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:01:30.399719 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:30.399690    4521 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c45pw\""
Oct 30 21:01:30.403914 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:30.403892    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f768d99c-rp29v"]
Oct 30 21:01:30.413725 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:30.413691    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfcd9487-cee0-4474-853f-29b908b88366-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f768d99c-rp29v\" (UID: \"dfcd9487-cee0-4474-853f-29b908b88366\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f768d99c-rp29v"
Oct 30 21:01:30.413899 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:30.413806    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z59d2\" (UniqueName: \"kubernetes.io/projected/dfcd9487-cee0-4474-853f-29b908b88366-kube-api-access-z59d2\") pod \"maas-api-revoke-sub-f768d99c-rp29v\" (UID: \"dfcd9487-cee0-4474-853f-29b908b88366\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f768d99c-rp29v"
Oct 30 21:01:30.413899 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:30.413849    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfcd9487-cee0-4474-853f-29b908b88366-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f768d99c-rp29v\" (UID: \"dfcd9487-cee0-4474-853f-29b908b88366\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f768d99c-rp29v"
Oct 30 21:01:30.515084 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:30.515047    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z59d2\" (UniqueName: \"kubernetes.io/projected/dfcd9487-cee0-4474-853f-29b908b88366-kube-api-access-z59d2\") pod \"maas-api-revoke-sub-f768d99c-rp29v\" (UID: \"dfcd9487-cee0-4474-853f-29b908b88366\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f768d99c-rp29v"
Oct 30 21:01:30.515282 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:30.515130    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfcd9487-cee0-4474-853f-29b908b88366-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f768d99c-rp29v\" (UID: \"dfcd9487-cee0-4474-853f-29b908b88366\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f768d99c-rp29v"
Oct 30 21:01:30.515282 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:30.515261    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfcd9487-cee0-4474-853f-29b908b88366-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f768d99c-rp29v\" (UID: \"dfcd9487-cee0-4474-853f-29b908b88366\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f768d99c-rp29v"
Oct 30 21:01:30.516178 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:30.516136    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfcd9487-cee0-4474-853f-29b908b88366-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f768d99c-rp29v\" (UID: \"dfcd9487-cee0-4474-853f-29b908b88366\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f768d99c-rp29v"
Oct 30 21:01:30.520731 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:30.520668    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfcd9487-cee0-4474-853f-29b908b88366-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f768d99c-rp29v\" (UID: \"dfcd9487-cee0-4474-853f-29b908b88366\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f768d99c-rp29v"
Oct 30 21:01:30.540594 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:30.539098    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z59d2\" (UniqueName: \"kubernetes.io/projected/dfcd9487-cee0-4474-853f-29b908b88366-kube-api-access-z59d2\") pod \"maas-api-revoke-sub-f768d99c-rp29v\" (UID: \"dfcd9487-cee0-4474-853f-29b908b88366\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f768d99c-rp29v"
Oct 30 21:01:30.713472 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:30.713427    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f768d99c-rp29v"
Oct 30 21:01:31.120269 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:31.105127    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f768d99c-rp29v"]
Oct 30 21:01:31.120269 ip-10-0-131-125 kubenswrapper[4521]: W1030 21:01:31.114569    4521 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfcd9487_cee0_4474_853f_29b908b88366.slice/crio-bf783282bbfcd217267b32efc593a7c4303fa6fe64055922ea45f1cdb603fe8b WatchSource:0}: Error finding container bf783282bbfcd217267b32efc593a7c4303fa6fe64055922ea45f1cdb603fe8b: Status 404 returned error can't find the container with id bf783282bbfcd217267b32efc593a7c4303fa6fe64055922ea45f1cdb603fe8b
Oct 30 21:01:31.320098 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:31.320046    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f768d99c-rp29v" event={"ID":"dfcd9487-cee0-4474-853f-29b908b88366","Type":"ContainerStarted","Data":"bf783282bbfcd217267b32efc593a7c4303fa6fe64055922ea45f1cdb603fe8b"}
Oct 30 21:01:32.364562 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:32.361152    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f768d99c-rp29v" event={"ID":"dfcd9487-cee0-4474-853f-29b908b88366","Type":"ContainerStarted","Data":"8dbb906f9eeaf3f6deafd55bfc955340c0295d9dcf47ba601fba1c476c89fec6"}
Oct 30 21:01:32.773350 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:32.772540    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a0c5b/premium-w0-4a0c5b-kserve-85dcf89f57-mscdn" podUID="00e42758-77ce-4663-adfa-2202c0aae24a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:01:33.082411 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:33.081418    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-32a969/distinct-w2-32a969-kserve-7f798b664f-jdh82" podUID="887f1d27-bf24-4b57-b6e4-8fb095df7a50" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:01:33.103473 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:33.103426    4521 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-4a0c5b/premium-w0-4a0c5b-kserve-85dcf89f57-mscdn" podUID="00e42758-77ce-4663-adfa-2202c0aae24a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:01:33.126359 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:33.124639    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a8f94/distinct-w3-5a8f94-kserve-5f8b544d59-mf95j" podUID="b8ac1cca-4ebd-474c-b32f-a3595a4cd847" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded"
Oct 30 21:01:33.215347 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:33.214430    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c2d1eb/premium-w4-c2d1eb-kserve-8696fbd9b9-mxqhm" podUID="4bb0f898-b77f-419d-bff4-6413e82ef51d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:01:33.219347 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:33.215667    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-65968b/premium-w5-65968b-kserve-67977485f7-c9jpv" podUID="92b2dd65-8a83-436d-afa8-50e5f0d11fa0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:01:33.219347 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:33.215987    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a8f94/distinct-2-w3-5a8f94-kserve-ffcc6449b-hv48k" podUID="bc0f5352-5c44-4c35-adf5-3fffab06b644" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": context deadline exceeded"
Oct 30 21:01:33.282348 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:33.267045    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c4b379/distinct-2-w1-c4b379-kserve-7999b785f5-97fz5" podUID="080a654d-a5bf-4845-a07f-cc8a7e6d0ace" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": context deadline exceeded"
Oct 30 21:01:33.365073 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:33.361533    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-39fb96/premium-w6-39fb96-kserve-b9cf7f75c-96blx" podUID="18ca0a94-c322-41fa-9fd9-0f14419b5075" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded"
Oct 30 21:01:33.446398 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:33.420433    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a8f94/simulator-w3-5a8f94-kserve-77cc88cc89-n4sl6" podUID="1fa6900c-eee2-4f00-b0ec-248def48dfc4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 30 21:01:33.447213 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:33.446744    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-32a969/simulator-w2-32a969-kserve-6cc8dfc44c-dwhlt" podUID="62a6adf7-d17a-4fec-a707-a2d549b67753" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:01:33.452493 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:33.451854    4521 generic.go:356] "Generic (PLEG): container finished" podID="dfcd9487-cee0-4474-853f-29b908b88366" containerID="8dbb906f9eeaf3f6deafd55bfc955340c0295d9dcf47ba601fba1c476c89fec6" exitCode=0
Oct 30 21:01:33.452493 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:33.452064    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f768d99c-rp29v" event={"ID":"dfcd9487-cee0-4474-853f-29b908b88366","Type":"ContainerDied","Data":"8dbb906f9eeaf3f6deafd55bfc955340c0295d9dcf47ba601fba1c476c89fec6"}
Oct 30 21:01:33.467666 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:33.467593    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a8f94/premium-w3-5a8f94-kserve-bb96b6848-5hk8h" podUID="24ccc37d-027a-4097-8da5-04df417aef94" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:01:33.708962 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:33.707788    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-32a969/distinct-2-w2-32a969-kserve-5c7845448b-vt5gw" podUID="d430ed01-65d1-43a6-a0c4-ab82212376a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:01:33.714395 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:33.710622    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c4b379/premium-w1-c4b379-kserve-575f896d49-2n6gz" podUID="2aac3e8e-d631-4b9c-bdbb-3b4b1df133a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:01:33.793215 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:33.793152    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a0c5b/premium-w0-4a0c5b-kserve-85dcf89f57-mscdn" podUID="00e42758-77ce-4663-adfa-2202c0aae24a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:01:33.827374 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:33.818395    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4a0c5b/premium-w0-4a0c5b-kserve-85dcf89f57-mscdn"
Oct 30 21:01:33.854660 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:33.854624    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4a0c5b/premium-w0-4a0c5b-kserve-85dcf89f57-mscdn"
Oct 30 21:01:34.137402 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:34.127839    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a8f94/distinct-w3-5a8f94-kserve-5f8b544d59-mf95j" podUID="b8ac1cca-4ebd-474c-b32f-a3595a4cd847" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:01:34.137402 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:34.128501    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5a8f94/distinct-w3-5a8f94-kserve-5f8b544d59-mf95j"
Oct 30 21:01:34.152732 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:34.152229    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5a8f94/distinct-w3-5a8f94-kserve-5f8b544d59-mf95j"
Oct 30 21:01:34.371580 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:34.371530    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-39fb96/premium-w6-39fb96-kserve-b9cf7f75c-96blx" podUID="18ca0a94-c322-41fa-9fd9-0f14419b5075" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:01:34.496455 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:34.448505    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-32a969/simulator-w2-32a969-kserve-6cc8dfc44c-dwhlt" podUID="62a6adf7-d17a-4fec-a707-a2d549b67753" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:01:34.496455 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:34.379735    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-65968b/premium-w5-65968b-kserve-67977485f7-c9jpv" podUID="92b2dd65-8a83-436d-afa8-50e5f0d11fa0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 30 21:01:34.496455 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:34.450476    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-39fb96/premium-w6-39fb96-kserve-b9cf7f75c-96blx"
Oct 30 21:01:34.496455 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:34.450601    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-32a969/simulator-w2-32a969-kserve-6cc8dfc44c-dwhlt"
Oct 30 21:01:34.496455 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:34.450688    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-65968b/premium-w5-65968b-kserve-67977485f7-c9jpv"
Oct 30 21:01:34.496455 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:34.379775    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a8f94/distinct-2-w3-5a8f94-kserve-ffcc6449b-hv48k" podUID="bc0f5352-5c44-4c35-adf5-3fffab06b644" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": context deadline exceeded"
Oct 30 21:01:34.496455 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:34.379816    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c4b379/distinct-2-w1-c4b379-kserve-7999b785f5-97fz5" podUID="080a654d-a5bf-4845-a07f-cc8a7e6d0ace" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": context deadline exceeded"
Oct 30 21:01:34.496455 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:34.470031    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a8f94/premium-w3-5a8f94-kserve-bb96b6848-5hk8h" podUID="24ccc37d-027a-4097-8da5-04df417aef94" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Oct 30 21:01:34.496455 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:34.473613    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5a8f94/distinct-2-w3-5a8f94-kserve-ffcc6449b-hv48k"
Oct 30 21:01:34.496455 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:34.473706    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c4b379/distinct-2-w1-c4b379-kserve-7999b785f5-97fz5"
Oct 30 21:01:34.496455 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:34.473765    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5a8f94/premium-w3-5a8f94-kserve-bb96b6848-5hk8h"
Oct 30 21:01:34.496455 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:34.432535    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a8f94/simulator-w3-5a8f94-kserve-77cc88cc89-n4sl6" podUID="1fa6900c-eee2-4f00-b0ec-248def48dfc4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:01:34.518348 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:34.501665    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5a8f94/simulator-w3-5a8f94-kserve-77cc88cc89-n4sl6"
Oct 30 21:01:34.743377 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:34.732162    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-32a969/distinct-2-w2-32a969-kserve-5c7845448b-vt5gw" podUID="d430ed01-65d1-43a6-a0c4-ab82212376a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded"
Oct 30 21:01:34.746183 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:34.746151    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-32a969/distinct-2-w2-32a969-kserve-5c7845448b-vt5gw"
Oct 30 21:01:34.886412 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:34.882768    4521 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-c4b379/distinct-2-w1-c4b379-kserve-7999b785f5-97fz5" podUID="080a654d-a5bf-4845-a07f-cc8a7e6d0ace" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:01:35.095123 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:35.089510    4521 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-32a969/simulator-w2-32a969-kserve-6cc8dfc44c-dwhlt" podUID="62a6adf7-d17a-4fec-a707-a2d549b67753" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": context deadline exceeded"
Oct 30 21:01:35.160871 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:35.159514    4521 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-5a8f94/distinct-w3-5a8f94-kserve-5f8b544d59-mf95j" podUID="b8ac1cca-4ebd-474c-b32f-a3595a4cd847" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/health\": context deadline exceeded"
Oct 30 21:01:35.162400 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:35.161469    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a8f94/distinct-w3-5a8f94-kserve-5f8b544d59-mf95j" podUID="b8ac1cca-4ebd-474c-b32f-a3595a4cd847" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:01:35.318369 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:35.313392    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-32a969/distinct-2-w2-32a969-kserve-5c7845448b-vt5gw"
Oct 30 21:01:35.390497 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:35.390446    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c2d1eb/premium-w4-c2d1eb-kserve-8696fbd9b9-mxqhm" podUID="4bb0f898-b77f-419d-bff4-6413e82ef51d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded"
Oct 30 21:01:35.432528 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:35.432430    4521 patch_prober.go:29] interesting pod/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-f4lmh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:01:35.432758 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:35.432732    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-f4lmh" podUID="9cb1b1d5-a595-403d-9a53-0560efac6ab7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:01:35.446281 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:35.446240    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-39fb96/premium-w6-39fb96-kserve-b9cf7f75c-96blx" podUID="18ca0a94-c322-41fa-9fd9-0f14419b5075" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:01:35.462749 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:35.462696    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c4b379/distinct-2-w1-c4b379-kserve-7999b785f5-97fz5" podUID="080a654d-a5bf-4845-a07f-cc8a7e6d0ace" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:01:35.463961 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:35.462996    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-32a969/simulator-w2-32a969-kserve-6cc8dfc44c-dwhlt" podUID="62a6adf7-d17a-4fec-a707-a2d549b67753" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Oct 30 21:01:35.466159 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:35.463029    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-65968b/premium-w5-65968b-kserve-67977485f7-c9jpv" podUID="92b2dd65-8a83-436d-afa8-50e5f0d11fa0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:01:35.471346 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:35.463062    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a8f94/distinct-2-w3-5a8f94-kserve-ffcc6449b-hv48k" podUID="bc0f5352-5c44-4c35-adf5-3fffab06b644" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": context deadline exceeded"
Oct 30 21:01:35.480712 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:35.471996    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a8f94/premium-w3-5a8f94-kserve-bb96b6848-5hk8h" podUID="24ccc37d-027a-4097-8da5-04df417aef94" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:01:35.482027 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:35.480909    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a8f94/simulator-w3-5a8f94-kserve-77cc88cc89-n4sl6" podUID="1fa6900c-eee2-4f00-b0ec-248def48dfc4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:01:35.580166 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:35.580138    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5a8f94/distinct-2-w3-5a8f94-kserve-ffcc6449b-hv48k"
Oct 30 21:01:35.599760 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:35.599730    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5a8f94/premium-w3-5a8f94-kserve-bb96b6848-5hk8h"
Oct 30 21:01:35.738671 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:35.734509    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c4b379/premium-w1-c4b379-kserve-575f896d49-2n6gz" podUID="2aac3e8e-d631-4b9c-bdbb-3b4b1df133a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:01:35.855655 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:35.855608    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a0c5b/premium-w0-4a0c5b-kserve-85dcf89f57-mscdn" podUID="00e42758-77ce-4663-adfa-2202c0aae24a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:01:36.126717 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:36.114251    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-32a969/distinct-w2-32a969-kserve-7f798b664f-jdh82" podUID="887f1d27-bf24-4b57-b6e4-8fb095df7a50" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:01:36.211583 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:36.169701    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a8f94/distinct-w3-5a8f94-kserve-5f8b544d59-mf95j" podUID="b8ac1cca-4ebd-474c-b32f-a3595a4cd847" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:01:36.211583 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:36.171220    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5a8f94/distinct-w3-5a8f94-kserve-5f8b544d59-mf95j"
Oct 30 21:01:36.319052 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:36.318964    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f768d99c-rp29v"
Oct 30 21:01:36.442346 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:36.439314    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c2d1eb/premium-w4-c2d1eb-kserve-8696fbd9b9-mxqhm" podUID="4bb0f898-b77f-419d-bff4-6413e82ef51d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:01:36.442346 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:36.440066    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c2d1eb/premium-w4-c2d1eb-kserve-8696fbd9b9-mxqhm"
Oct 30 21:01:36.476366 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:36.466448    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-32a969/simulator-w2-32a969-kserve-6cc8dfc44c-dwhlt" podUID="62a6adf7-d17a-4fec-a707-a2d549b67753" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Oct 30 21:01:36.476366 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:36.466851    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-65968b/premium-w5-65968b-kserve-67977485f7-c9jpv" podUID="92b2dd65-8a83-436d-afa8-50e5f0d11fa0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:01:36.476366 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:36.467131    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-39fb96/premium-w6-39fb96-kserve-b9cf7f75c-96blx" podUID="18ca0a94-c322-41fa-9fd9-0f14419b5075" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded"
Oct 30 21:01:36.476366 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:36.467400    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c4b379/distinct-2-w1-c4b379-kserve-7999b785f5-97fz5" podUID="080a654d-a5bf-4845-a07f-cc8a7e6d0ace" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": context deadline exceeded"
Oct 30 21:01:36.478299 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:36.478268    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfcd9487-cee0-4474-853f-29b908b88366-kube-api-access-z59d2\" (UniqueName: \"kubernetes.io/projected/dfcd9487-cee0-4474-853f-29b908b88366-kube-api-access-z59d2\") pod \"dfcd9487-cee0-4474-853f-29b908b88366\" (UID: \"dfcd9487-cee0-4474-853f-29b908b88366\") "
Oct 30 21:01:36.478661 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:36.478640    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dfcd9487-cee0-4474-853f-29b908b88366-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfcd9487-cee0-4474-853f-29b908b88366-maas-api-openshift-ca\") pod \"dfcd9487-cee0-4474-853f-29b908b88366\" (UID: \"dfcd9487-cee0-4474-853f-29b908b88366\") "
Oct 30 21:01:36.478856 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:36.478841    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dfcd9487-cee0-4474-853f-29b908b88366-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfcd9487-cee0-4474-853f-29b908b88366-maas-api-xks-ca\") pod \"dfcd9487-cee0-4474-853f-29b908b88366\" (UID: \"dfcd9487-cee0-4474-853f-29b908b88366\") "
Oct 30 21:01:36.499532 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:36.499413    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a8f94/simulator-w3-5a8f94-kserve-77cc88cc89-n4sl6" podUID="1fa6900c-eee2-4f00-b0ec-248def48dfc4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:01:36.523362 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:36.517552    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfcd9487-cee0-4474-853f-29b908b88366-maas-api-openshift-ca" pod "dfcd9487-cee0-4474-853f-29b908b88366" (UID: "dfcd9487-cee0-4474-853f-29b908b88366"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:01:36.566210 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:36.566153    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfcd9487-cee0-4474-853f-29b908b88366-kube-api-access-z59d2" pod "dfcd9487-cee0-4474-853f-29b908b88366" (UID: "dfcd9487-cee0-4474-853f-29b908b88366"). InnerVolumeSpecName "kube-api-access-z59d2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:01:36.581723 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:36.581565    4521 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfcd9487-cee0-4474-853f-29b908b88366-maas-api-openshift-ca\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:01:36.581951 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:36.581931    4521 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z59d2\" (UniqueName: \"kubernetes.io/projected/dfcd9487-cee0-4474-853f-29b908b88366-kube-api-access-z59d2\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:01:36.659388 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:36.651676    4521 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-65968b/distinct-2-w5-65968b-kserve-6d65c85f6c-dz5r7"]
Oct 30 21:01:36.659388 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:36.652486    4521 cpu_manager.go:414] "containerMap: removing container" podUID="dfcd9487-cee0-4474-853f-29b908b88366" containerName="revoke-keys"
Oct 30 21:01:36.659388 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:36.652506    4521 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfcd9487-cee0-4474-853f-29b908b88366" containerName="revoke-keys"
Oct 30 21:01:36.659388 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:36.652720    4521 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfcd9487-cee0-4474-853f-29b908b88366" containerName="revoke-keys"
Oct 30 21:01:36.696372 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:36.668680    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-65968b/distinct-2-w5-65968b-kserve-6d65c85f6c-dz5r7"]
Oct 30 21:01:36.696707 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:36.696675    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f768d99c-rp29v" event={"ID":"dfcd9487-cee0-4474-853f-29b908b88366","Type":"ContainerDied","Data":"bf783282bbfcd217267b32efc593a7c4303fa6fe64055922ea45f1cdb603fe8b"}
Oct 30 21:01:36.696875 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:36.696857    4521 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bf783282bbfcd217267b32efc593a7c4303fa6fe64055922ea45f1cdb603fe8b"
Oct 30 21:01:36.696976 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:36.687356    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f768d99c-rp29v"
Oct 30 21:01:36.697345 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:36.671547    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-65968b/distinct-2-w5-65968b-kserve-6d65c85f6c-dz5r7"
Oct 30 21:01:36.703591 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:36.683030    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vjbg\" (UniqueName: \"kubernetes.io/projected/d6751cf5-7df8-4ffa-b6c8-8b0c4f16ba2a-kube-api-access-7vjbg\") pod \"distinct-2-w5-65968b-kserve-6d65c85f6c-dz5r7\" (UID: \"d6751cf5-7df8-4ffa-b6c8-8b0c4f16ba2a\") " pod="e2e-models-e2e-worker-w5-65968b/distinct-2-w5-65968b-kserve-6d65c85f6c-dz5r7"
Oct 30 21:01:36.704101 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:36.704080    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6751cf5-7df8-4ffa-b6c8-8b0c4f16ba2a-tmp-dir\") pod \"distinct-2-w5-65968b-kserve-6d65c85f6c-dz5r7\" (UID: \"d6751cf5-7df8-4ffa-b6c8-8b0c4f16ba2a\") " pod="e2e-models-e2e-worker-w5-65968b/distinct-2-w5-65968b-kserve-6d65c85f6c-dz5r7"
Oct 30 21:01:36.704671 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:36.704652    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6751cf5-7df8-4ffa-b6c8-8b0c4f16ba2a-home\") pod \"distinct-2-w5-65968b-kserve-6d65c85f6c-dz5r7\" (UID: \"d6751cf5-7df8-4ffa-b6c8-8b0c4f16ba2a\") " pod="e2e-models-e2e-worker-w5-65968b/distinct-2-w5-65968b-kserve-6d65c85f6c-dz5r7"
Oct 30 21:01:36.704788 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:36.704773    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6751cf5-7df8-4ffa-b6c8-8b0c4f16ba2a-dshm\") pod \"distinct-2-w5-65968b-kserve-6d65c85f6c-dz5r7\" (UID: \"d6751cf5-7df8-4ffa-b6c8-8b0c4f16ba2a\") " pod="e2e-models-e2e-worker-w5-65968b/distinct-2-w5-65968b-kserve-6d65c85f6c-dz5r7"
Oct 30 21:01:36.705080 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:36.704906    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6751cf5-7df8-4ffa-b6c8-8b0c4f16ba2a-model-cache\") pod \"distinct-2-w5-65968b-kserve-6d65c85f6c-dz5r7\" (UID: \"d6751cf5-7df8-4ffa-b6c8-8b0c4f16ba2a\") " pod="e2e-models-e2e-worker-w5-65968b/distinct-2-w5-65968b-kserve-6d65c85f6c-dz5r7"
Oct 30 21:01:36.705704 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:36.705679    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6751cf5-7df8-4ffa-b6c8-8b0c4f16ba2a-tls-certs\") pod \"distinct-2-w5-65968b-kserve-6d65c85f6c-dz5r7\" (UID: \"d6751cf5-7df8-4ffa-b6c8-8b0c4f16ba2a\") " pod="e2e-models-e2e-worker-w5-65968b/distinct-2-w5-65968b-kserve-6d65c85f6c-dz5r7"
Oct 30 21:01:36.731344 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:36.728625    4521 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-65968b\"/\"distinct-2-w5-65968b-kserve-self-signed-certs\""
Oct 30 21:01:36.737339 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:36.732463    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dfcd9487-cee0-4474-853f-29b908b88366-maas-api-xks-ca" pod "dfcd9487-cee0-4474-853f-29b908b88366" (UID: "dfcd9487-cee0-4474-853f-29b908b88366"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:01:36.751475 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:36.750992    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c4b379/premium-w1-c4b379-kserve-575f896d49-2n6gz" podUID="2aac3e8e-d631-4b9c-bdbb-3b4b1df133a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:01:36.755953 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:36.755925    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c4b379/premium-w1-c4b379-kserve-575f896d49-2n6gz"
Oct 30 21:01:36.860407 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:36.841418    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6751cf5-7df8-4ffa-b6c8-8b0c4f16ba2a-tmp-dir\") pod \"distinct-2-w5-65968b-kserve-6d65c85f6c-dz5r7\" (UID: \"d6751cf5-7df8-4ffa-b6c8-8b0c4f16ba2a\") " pod="e2e-models-e2e-worker-w5-65968b/distinct-2-w5-65968b-kserve-6d65c85f6c-dz5r7"
Oct 30 21:01:36.860407 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:36.841486    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6751cf5-7df8-4ffa-b6c8-8b0c4f16ba2a-home\") pod \"distinct-2-w5-65968b-kserve-6d65c85f6c-dz5r7\" (UID: \"d6751cf5-7df8-4ffa-b6c8-8b0c4f16ba2a\") " pod="e2e-models-e2e-worker-w5-65968b/distinct-2-w5-65968b-kserve-6d65c85f6c-dz5r7"
Oct 30 21:01:36.860407 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:36.841516    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6751cf5-7df8-4ffa-b6c8-8b0c4f16ba2a-dshm\") pod \"distinct-2-w5-65968b-kserve-6d65c85f6c-dz5r7\" (UID: \"d6751cf5-7df8-4ffa-b6c8-8b0c4f16ba2a\") " pod="e2e-models-e2e-worker-w5-65968b/distinct-2-w5-65968b-kserve-6d65c85f6c-dz5r7"
Oct 30 21:01:36.860407 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:36.841566    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6751cf5-7df8-4ffa-b6c8-8b0c4f16ba2a-model-cache\") pod \"distinct-2-w5-65968b-kserve-6d65c85f6c-dz5r7\" (UID: \"d6751cf5-7df8-4ffa-b6c8-8b0c4f16ba2a\") " pod="e2e-models-e2e-worker-w5-65968b/distinct-2-w5-65968b-kserve-6d65c85f6c-dz5r7"
Oct 30 21:01:36.860407 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:36.841604    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6751cf5-7df8-4ffa-b6c8-8b0c4f16ba2a-tls-certs\") pod \"distinct-2-w5-65968b-kserve-6d65c85f6c-dz5r7\" (UID: \"d6751cf5-7df8-4ffa-b6c8-8b0c4f16ba2a\") " pod="e2e-models-e2e-worker-w5-65968b/distinct-2-w5-65968b-kserve-6d65c85f6c-dz5r7"
Oct 30 21:01:36.860407 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:36.841836    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vjbg\" (UniqueName: \"kubernetes.io/projected/d6751cf5-7df8-4ffa-b6c8-8b0c4f16ba2a-kube-api-access-7vjbg\") pod \"distinct-2-w5-65968b-kserve-6d65c85f6c-dz5r7\" (UID: \"d6751cf5-7df8-4ffa-b6c8-8b0c4f16ba2a\") " pod="e2e-models-e2e-worker-w5-65968b/distinct-2-w5-65968b-kserve-6d65c85f6c-dz5r7"
Oct 30 21:01:36.860407 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:36.842007    4521 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfcd9487-cee0-4474-853f-29b908b88366-maas-api-xks-ca\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:01:36.860407 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:36.842855    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6751cf5-7df8-4ffa-b6c8-8b0c4f16ba2a-tmp-dir\") pod \"distinct-2-w5-65968b-kserve-6d65c85f6c-dz5r7\" (UID: \"d6751cf5-7df8-4ffa-b6c8-8b0c4f16ba2a\") " pod="e2e-models-e2e-worker-w5-65968b/distinct-2-w5-65968b-kserve-6d65c85f6c-dz5r7"
Oct 30 21:01:36.860407 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:36.843132    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6751cf5-7df8-4ffa-b6c8-8b0c4f16ba2a-home\") pod \"distinct-2-w5-65968b-kserve-6d65c85f6c-dz5r7\" (UID: \"d6751cf5-7df8-4ffa-b6c8-8b0c4f16ba2a\") " pod="e2e-models-e2e-worker-w5-65968b/distinct-2-w5-65968b-kserve-6d65c85f6c-dz5r7"
Oct 30 21:01:36.926371 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:36.868494    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a0c5b/premium-w0-4a0c5b-kserve-85dcf89f57-mscdn" podUID="00e42758-77ce-4663-adfa-2202c0aae24a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:01:36.926371 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:36.870381    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6751cf5-7df8-4ffa-b6c8-8b0c4f16ba2a-model-cache\") pod \"distinct-2-w5-65968b-kserve-6d65c85f6c-dz5r7\" (UID: \"d6751cf5-7df8-4ffa-b6c8-8b0c4f16ba2a\") " pod="e2e-models-e2e-worker-w5-65968b/distinct-2-w5-65968b-kserve-6d65c85f6c-dz5r7"
Oct 30 21:01:36.926371 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:36.872600    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6751cf5-7df8-4ffa-b6c8-8b0c4f16ba2a-dshm\") pod \"distinct-2-w5-65968b-kserve-6d65c85f6c-dz5r7\" (UID: \"d6751cf5-7df8-4ffa-b6c8-8b0c4f16ba2a\") " pod="e2e-models-e2e-worker-w5-65968b/distinct-2-w5-65968b-kserve-6d65c85f6c-dz5r7"
Oct 30 21:01:36.926371 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:36.922926    4521 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-c2d1eb/premium-w4-c2d1eb-kserve-8696fbd9b9-mxqhm" podUID="4bb0f898-b77f-419d-bff4-6413e82ef51d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:01:36.999345 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:36.999298    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6751cf5-7df8-4ffa-b6c8-8b0c4f16ba2a-tls-certs\") pod \"distinct-2-w5-65968b-kserve-6d65c85f6c-dz5r7\" (UID: \"d6751cf5-7df8-4ffa-b6c8-8b0c4f16ba2a\") " pod="e2e-models-e2e-worker-w5-65968b/distinct-2-w5-65968b-kserve-6d65c85f6c-dz5r7"
Oct 30 21:01:37.017397 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:37.001186    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4a0c5b/premium-w0-4a0c5b-kserve-85dcf89f57-mscdn"
Oct 30 21:01:37.017397 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:37.014502    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-65968b/premium-w5-65968b-kserve-67977485f7-c9jpv"
Oct 30 21:01:37.017397 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:37.014542    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-32a969/simulator-w2-32a969-kserve-6cc8dfc44c-dwhlt"
Oct 30 21:01:37.017397 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:37.014573    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4a0c5b/premium-w0-4a0c5b-kserve-85dcf89f57-mscdn"
Oct 30 21:01:37.017397 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:37.014604    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5a8f94/simulator-w3-5a8f94-kserve-77cc88cc89-n4sl6"
Oct 30 21:01:37.065740 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:37.065670    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c4b379/premium-w1-c4b379-kserve-575f896d49-2n6gz"
Oct 30 21:01:37.098586 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:37.094206    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vjbg\" (UniqueName: \"kubernetes.io/projected/d6751cf5-7df8-4ffa-b6c8-8b0c4f16ba2a-kube-api-access-7vjbg\") pod \"distinct-2-w5-65968b-kserve-6d65c85f6c-dz5r7\" (UID: \"d6751cf5-7df8-4ffa-b6c8-8b0c4f16ba2a\") " pod="e2e-models-e2e-worker-w5-65968b/distinct-2-w5-65968b-kserve-6d65c85f6c-dz5r7"
Oct 30 21:01:37.110652 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:37.106823    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c2d1eb/premium-w4-c2d1eb-kserve-8696fbd9b9-mxqhm"
Oct 30 21:01:37.173443 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:37.171579    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5a8f94/distinct-w3-5a8f94-kserve-5f8b544d59-mf95j"
Oct 30 21:01:37.173443 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:37.172277    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-39fb96/premium-w6-39fb96-kserve-b9cf7f75c-96blx"
Oct 30 21:01:37.175316 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:37.174635    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c4b379/distinct-2-w1-c4b379-kserve-7999b785f5-97fz5"
Oct 30 21:01:37.359567 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:37.359296    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-65968b/distinct-2-w5-65968b-kserve-6d65c85f6c-dz5r7"
Oct 30 21:01:37.661801 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:37.661766    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-65968b/distinct-2-w5-65968b-kserve-6d65c85f6c-dz5r7"]
Oct 30 21:01:37.662203 ip-10-0-131-125 kubenswrapper[4521]: W1030 21:01:37.662032    4521 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd6751cf5_7df8_4ffa_b6c8_8b0c4f16ba2a.slice/crio-ddcc10eae2773392a1659a2c8896257485ab56351be3d2d1e78fe797306209f0 WatchSource:0}: Error finding container ddcc10eae2773392a1659a2c8896257485ab56351be3d2d1e78fe797306209f0: Status 404 returned error can't find the container with id ddcc10eae2773392a1659a2c8896257485ab56351be3d2d1e78fe797306209f0
Oct 30 21:01:37.698865 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:37.698828    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-65968b/distinct-2-w5-65968b-kserve-6d65c85f6c-dz5r7" event={"ID":"d6751cf5-7df8-4ffa-b6c8-8b0c4f16ba2a","Type":"ContainerStarted","Data":"ddcc10eae2773392a1659a2c8896257485ab56351be3d2d1e78fe797306209f0"}
Oct 30 21:01:37.896638 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:37.896603    4521 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6880f37-l42pm"]
Oct 30 21:01:37.900911 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:37.900885    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6880f37-l42pm"
Oct 30 21:01:37.905268 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:37.905238    4521 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c45pw\""
Oct 30 21:01:37.905421 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:37.905313    4521 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:01:37.911313 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:37.910029    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6880f37-l42pm"]
Oct 30 21:01:37.979919 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:37.979886    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/121ec831-c071-4e2e-a05c-24f4aa974505-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a6880f37-l42pm\" (UID: \"121ec831-c071-4e2e-a05c-24f4aa974505\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6880f37-l42pm"
Oct 30 21:01:37.980081 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:37.980040    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/121ec831-c071-4e2e-a05c-24f4aa974505-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a6880f37-l42pm\" (UID: \"121ec831-c071-4e2e-a05c-24f4aa974505\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6880f37-l42pm"
Oct 30 21:01:37.980140 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:37.980086    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpm2l\" (UniqueName: \"kubernetes.io/projected/121ec831-c071-4e2e-a05c-24f4aa974505-kube-api-access-kpm2l\") pod \"maas-api-revoke-sub-a6880f37-l42pm\" (UID: \"121ec831-c071-4e2e-a05c-24f4aa974505\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6880f37-l42pm"
Oct 30 21:01:38.080958 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:38.080926    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/121ec831-c071-4e2e-a05c-24f4aa974505-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a6880f37-l42pm\" (UID: \"121ec831-c071-4e2e-a05c-24f4aa974505\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6880f37-l42pm"
Oct 30 21:01:38.081124 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:38.080968    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpm2l\" (UniqueName: \"kubernetes.io/projected/121ec831-c071-4e2e-a05c-24f4aa974505-kube-api-access-kpm2l\") pod \"maas-api-revoke-sub-a6880f37-l42pm\" (UID: \"121ec831-c071-4e2e-a05c-24f4aa974505\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6880f37-l42pm"
Oct 30 21:01:38.081124 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:38.081097    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/121ec831-c071-4e2e-a05c-24f4aa974505-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a6880f37-l42pm\" (UID: \"121ec831-c071-4e2e-a05c-24f4aa974505\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6880f37-l42pm"
Oct 30 21:01:38.081837 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:38.081801    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/121ec831-c071-4e2e-a05c-24f4aa974505-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a6880f37-l42pm\" (UID: \"121ec831-c071-4e2e-a05c-24f4aa974505\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6880f37-l42pm"
Oct 30 21:01:38.083594 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:38.083566    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/121ec831-c071-4e2e-a05c-24f4aa974505-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a6880f37-l42pm\" (UID: \"121ec831-c071-4e2e-a05c-24f4aa974505\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6880f37-l42pm"
Oct 30 21:01:38.093362 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:38.093319    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpm2l\" (UniqueName: \"kubernetes.io/projected/121ec831-c071-4e2e-a05c-24f4aa974505-kube-api-access-kpm2l\") pod \"maas-api-revoke-sub-a6880f37-l42pm\" (UID: \"121ec831-c071-4e2e-a05c-24f4aa974505\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6880f37-l42pm"
Oct 30 21:01:38.264794 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:38.264759    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6880f37-l42pm"
Oct 30 21:01:38.564025 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:38.563994    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6880f37-l42pm"]
Oct 30 21:01:38.567448 ip-10-0-131-125 kubenswrapper[4521]: W1030 21:01:38.567423    4521 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod121ec831_c071_4e2e_a05c_24f4aa974505.slice/crio-2863ab1359e5427702703683c77a761af86cfd9359fcb0888a4c2ff78dc73372 WatchSource:0}: Error finding container 2863ab1359e5427702703683c77a761af86cfd9359fcb0888a4c2ff78dc73372: Status 404 returned error can't find the container with id 2863ab1359e5427702703683c77a761af86cfd9359fcb0888a4c2ff78dc73372
Oct 30 21:01:38.708943 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:38.708902    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-65968b/distinct-2-w5-65968b-kserve-6d65c85f6c-dz5r7" event={"ID":"d6751cf5-7df8-4ffa-b6c8-8b0c4f16ba2a","Type":"ContainerStarted","Data":"7e134bd3e9ae8eb8c9af61c92d85f54bc8f6e034ab4f9fcbca50396793babf02"}
Oct 30 21:01:38.710470 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:38.710439    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6880f37-l42pm" event={"ID":"121ec831-c071-4e2e-a05c-24f4aa974505","Type":"ContainerStarted","Data":"2863ab1359e5427702703683c77a761af86cfd9359fcb0888a4c2ff78dc73372"}
Oct 30 21:01:38.729961 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:38.729893    4521 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-65968b/distinct-2-w5-65968b-kserve-6d65c85f6c-dz5r7" podStartSLOduration=2.729874411 podStartE2EDuration="2.729874411s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:01:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:01:38.728145617 +0000 UTC m=+1366.073766882" watchObservedRunningTime="2026-10-30 21:01:38.729874411 +0000 UTC m=+1366.075495650"
Oct 30 21:01:38.912561 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:38.911264    4521 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bad53a71-964rp"]
Oct 30 21:01:38.938943 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:38.932273    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bad53a71-964rp"]
Oct 30 21:01:38.938943 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:38.932437    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bad53a71-964rp"
Oct 30 21:01:38.991105 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:38.991029    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phjr2\" (UniqueName: \"kubernetes.io/projected/19bae71e-c48b-4b51-9e5c-ea7ce7735057-kube-api-access-phjr2\") pod \"maas-api-revoke-sub-bad53a71-964rp\" (UID: \"19bae71e-c48b-4b51-9e5c-ea7ce7735057\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bad53a71-964rp"
Oct 30 21:01:38.991105 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:38.991096    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19bae71e-c48b-4b51-9e5c-ea7ce7735057-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bad53a71-964rp\" (UID: \"19bae71e-c48b-4b51-9e5c-ea7ce7735057\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bad53a71-964rp"
Oct 30 21:01:38.991341 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:38.991279    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19bae71e-c48b-4b51-9e5c-ea7ce7735057-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bad53a71-964rp\" (UID: \"19bae71e-c48b-4b51-9e5c-ea7ce7735057\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bad53a71-964rp"
Oct 30 21:01:39.092966 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:39.092878    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19bae71e-c48b-4b51-9e5c-ea7ce7735057-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bad53a71-964rp\" (UID: \"19bae71e-c48b-4b51-9e5c-ea7ce7735057\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bad53a71-964rp"
Oct 30 21:01:39.093115 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:39.093029    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phjr2\" (UniqueName: \"kubernetes.io/projected/19bae71e-c48b-4b51-9e5c-ea7ce7735057-kube-api-access-phjr2\") pod \"maas-api-revoke-sub-bad53a71-964rp\" (UID: \"19bae71e-c48b-4b51-9e5c-ea7ce7735057\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bad53a71-964rp"
Oct 30 21:01:39.093115 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:39.093069    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19bae71e-c48b-4b51-9e5c-ea7ce7735057-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bad53a71-964rp\" (UID: \"19bae71e-c48b-4b51-9e5c-ea7ce7735057\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bad53a71-964rp"
Oct 30 21:01:39.093913 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:39.093885    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19bae71e-c48b-4b51-9e5c-ea7ce7735057-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bad53a71-964rp\" (UID: \"19bae71e-c48b-4b51-9e5c-ea7ce7735057\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bad53a71-964rp"
Oct 30 21:01:39.095415 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:39.095386    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19bae71e-c48b-4b51-9e5c-ea7ce7735057-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bad53a71-964rp\" (UID: \"19bae71e-c48b-4b51-9e5c-ea7ce7735057\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bad53a71-964rp"
Oct 30 21:01:39.105566 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:39.105538    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phjr2\" (UniqueName: \"kubernetes.io/projected/19bae71e-c48b-4b51-9e5c-ea7ce7735057-kube-api-access-phjr2\") pod \"maas-api-revoke-sub-bad53a71-964rp\" (UID: \"19bae71e-c48b-4b51-9e5c-ea7ce7735057\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bad53a71-964rp"
Oct 30 21:01:39.272389 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:39.272354    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bad53a71-964rp"
Oct 30 21:01:39.361380 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:39.359822    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-65968b/distinct-2-w5-65968b-kserve-6d65c85f6c-dz5r7"
Oct 30 21:01:39.570646 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:39.570606    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bad53a71-964rp"]
Oct 30 21:01:39.579402 ip-10-0-131-125 kubenswrapper[4521]: W1030 21:01:39.579368    4521 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19bae71e_c48b_4b51_9e5c_ea7ce7735057.slice/crio-448dff48db294560e26702a976066bc247c76c589548fb6921c43e0a1b8836f6 WatchSource:0}: Error finding container 448dff48db294560e26702a976066bc247c76c589548fb6921c43e0a1b8836f6: Status 404 returned error can't find the container with id 448dff48db294560e26702a976066bc247c76c589548fb6921c43e0a1b8836f6
Oct 30 21:01:39.724734 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:39.724691    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bad53a71-964rp" event={"ID":"19bae71e-c48b-4b51-9e5c-ea7ce7735057","Type":"ContainerStarted","Data":"448dff48db294560e26702a976066bc247c76c589548fb6921c43e0a1b8836f6"}
Oct 30 21:01:39.726889 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:39.726858    4521 generic.go:356] "Generic (PLEG): container finished" podID="121ec831-c071-4e2e-a05c-24f4aa974505" containerID="d48ab49fd65886e8985a68162f760a23b8f74b1c53bd5d1137d534f014c66d85" exitCode=0
Oct 30 21:01:39.727047 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:39.726941    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6880f37-l42pm" event={"ID":"121ec831-c071-4e2e-a05c-24f4aa974505","Type":"ContainerDied","Data":"d48ab49fd65886e8985a68162f760a23b8f74b1c53bd5d1137d534f014c66d85"}
Oct 30 21:01:40.741139 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:40.741106    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bad53a71-964rp" event={"ID":"19bae71e-c48b-4b51-9e5c-ea7ce7735057","Type":"ContainerStarted","Data":"154f07e6bcb5d2c20bd1082b0e7eae54ff53298198e064d549d192043eca663b"}
Oct 30 21:01:40.907617 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:40.907586    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6880f37-l42pm"
Oct 30 21:01:40.917138 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:40.917103    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/121ec831-c071-4e2e-a05c-24f4aa974505-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/121ec831-c071-4e2e-a05c-24f4aa974505-maas-api-openshift-ca\") pod \"121ec831-c071-4e2e-a05c-24f4aa974505\" (UID: \"121ec831-c071-4e2e-a05c-24f4aa974505\") "
Oct 30 21:01:40.917385 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:40.917366    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/121ec831-c071-4e2e-a05c-24f4aa974505-kube-api-access-kpm2l\" (UniqueName: \"kubernetes.io/projected/121ec831-c071-4e2e-a05c-24f4aa974505-kube-api-access-kpm2l\") pod \"121ec831-c071-4e2e-a05c-24f4aa974505\" (UID: \"121ec831-c071-4e2e-a05c-24f4aa974505\") "
Oct 30 21:01:40.917483 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:40.917415    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/121ec831-c071-4e2e-a05c-24f4aa974505-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/121ec831-c071-4e2e-a05c-24f4aa974505-maas-api-xks-ca\") pod \"121ec831-c071-4e2e-a05c-24f4aa974505\" (UID: \"121ec831-c071-4e2e-a05c-24f4aa974505\") "
Oct 30 21:01:40.917540 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:40.917514    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/121ec831-c071-4e2e-a05c-24f4aa974505-maas-api-openshift-ca" pod "121ec831-c071-4e2e-a05c-24f4aa974505" (UID: "121ec831-c071-4e2e-a05c-24f4aa974505"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:01:40.918052 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:40.918018    4521 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/121ec831-c071-4e2e-a05c-24f4aa974505-maas-api-openshift-ca\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:01:40.921953 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:40.919972    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/121ec831-c071-4e2e-a05c-24f4aa974505-maas-api-xks-ca" pod "121ec831-c071-4e2e-a05c-24f4aa974505" (UID: "121ec831-c071-4e2e-a05c-24f4aa974505"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:01:40.921953 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:40.920317    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/121ec831-c071-4e2e-a05c-24f4aa974505-kube-api-access-kpm2l" pod "121ec831-c071-4e2e-a05c-24f4aa974505" (UID: "121ec831-c071-4e2e-a05c-24f4aa974505"). InnerVolumeSpecName "kube-api-access-kpm2l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:01:41.020063 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:41.019976    4521 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kpm2l\" (UniqueName: \"kubernetes.io/projected/121ec831-c071-4e2e-a05c-24f4aa974505-kube-api-access-kpm2l\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:01:41.020277 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:41.020248    4521 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/121ec831-c071-4e2e-a05c-24f4aa974505-maas-api-xks-ca\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:01:41.754248 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:41.754202    4521 generic.go:356] "Generic (PLEG): container finished" podID="19bae71e-c48b-4b51-9e5c-ea7ce7735057" containerID="154f07e6bcb5d2c20bd1082b0e7eae54ff53298198e064d549d192043eca663b" exitCode=0
Oct 30 21:01:41.754758 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:41.754418    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bad53a71-964rp" event={"ID":"19bae71e-c48b-4b51-9e5c-ea7ce7735057","Type":"ContainerDied","Data":"154f07e6bcb5d2c20bd1082b0e7eae54ff53298198e064d549d192043eca663b"}
Oct 30 21:01:41.756606 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:41.756575    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6880f37-l42pm"
Oct 30 21:01:41.756728 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:41.756641    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6880f37-l42pm" event={"ID":"121ec831-c071-4e2e-a05c-24f4aa974505","Type":"ContainerDied","Data":"2863ab1359e5427702703683c77a761af86cfd9359fcb0888a4c2ff78dc73372"}
Oct 30 21:01:41.756728 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:41.756703    4521 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2863ab1359e5427702703683c77a761af86cfd9359fcb0888a4c2ff78dc73372"
Oct 30 21:01:41.925460 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:41.925431    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bad53a71-964rp"
Oct 30 21:01:42.007909 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:42.007830    4521 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19cf1ad9-gkr8q"]
Oct 30 21:01:42.008624 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:42.008486    4521 cpu_manager.go:414] "containerMap: removing container" podUID="121ec831-c071-4e2e-a05c-24f4aa974505" containerName="revoke-keys"
Oct 30 21:01:42.008624 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:42.008507    4521 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="121ec831-c071-4e2e-a05c-24f4aa974505" containerName="revoke-keys"
Oct 30 21:01:42.008624 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:42.008538    4521 cpu_manager.go:414] "containerMap: removing container" podUID="19bae71e-c48b-4b51-9e5c-ea7ce7735057" containerName="revoke-keys"
Oct 30 21:01:42.008624 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:42.008543    4521 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19bae71e-c48b-4b51-9e5c-ea7ce7735057" containerName="revoke-keys"
Oct 30 21:01:42.008944 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:42.008696    4521 memory_manager.go:365] "RemoveStaleState removing state" podUID="19bae71e-c48b-4b51-9e5c-ea7ce7735057" containerName="revoke-keys"
Oct 30 21:01:42.008944 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:42.008711    4521 memory_manager.go:365] "RemoveStaleState removing state" podUID="121ec831-c071-4e2e-a05c-24f4aa974505" containerName="revoke-keys"
Oct 30 21:01:42.012122 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:42.012103    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19cf1ad9-gkr8q"
Oct 30 21:01:42.022899 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:42.022865    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19cf1ad9-gkr8q"]
Oct 30 21:01:42.048773 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:42.048738    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/19bae71e-c48b-4b51-9e5c-ea7ce7735057-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19bae71e-c48b-4b51-9e5c-ea7ce7735057-maas-api-xks-ca\") pod \"19bae71e-c48b-4b51-9e5c-ea7ce7735057\" (UID: \"19bae71e-c48b-4b51-9e5c-ea7ce7735057\") "
Oct 30 21:01:42.048952 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:42.048789    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19bae71e-c48b-4b51-9e5c-ea7ce7735057-kube-api-access-phjr2\" (UniqueName: \"kubernetes.io/projected/19bae71e-c48b-4b51-9e5c-ea7ce7735057-kube-api-access-phjr2\") pod \"19bae71e-c48b-4b51-9e5c-ea7ce7735057\" (UID: \"19bae71e-c48b-4b51-9e5c-ea7ce7735057\") "
Oct 30 21:01:42.049034 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:42.049008    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19bae71e-c48b-4b51-9e5c-ea7ce7735057-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19bae71e-c48b-4b51-9e5c-ea7ce7735057-maas-api-openshift-ca\") pod \"19bae71e-c48b-4b51-9e5c-ea7ce7735057\" (UID: \"19bae71e-c48b-4b51-9e5c-ea7ce7735057\") "
Oct 30 21:01:42.049230 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:42.049210    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc6e70c9-ff92-4d6b-bf03-28441c0750c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-19cf1ad9-gkr8q\" (UID: \"dc6e70c9-ff92-4d6b-bf03-28441c0750c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19cf1ad9-gkr8q"
Oct 30 21:01:42.049343 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:42.049250    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc6e70c9-ff92-4d6b-bf03-28441c0750c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-19cf1ad9-gkr8q\" (UID: \"dc6e70c9-ff92-4d6b-bf03-28441c0750c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19cf1ad9-gkr8q"
Oct 30 21:01:42.049423 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:42.049409    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfps8\" (UniqueName: \"kubernetes.io/projected/dc6e70c9-ff92-4d6b-bf03-28441c0750c3-kube-api-access-dfps8\") pod \"maas-api-revoke-sub-19cf1ad9-gkr8q\" (UID: \"dc6e70c9-ff92-4d6b-bf03-28441c0750c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19cf1ad9-gkr8q"
Oct 30 21:01:42.050182 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:42.050151    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19bae71e-c48b-4b51-9e5c-ea7ce7735057-maas-api-openshift-ca" pod "19bae71e-c48b-4b51-9e5c-ea7ce7735057" (UID: "19bae71e-c48b-4b51-9e5c-ea7ce7735057"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:01:42.052777 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:42.052736    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19bae71e-c48b-4b51-9e5c-ea7ce7735057-kube-api-access-phjr2" pod "19bae71e-c48b-4b51-9e5c-ea7ce7735057" (UID: "19bae71e-c48b-4b51-9e5c-ea7ce7735057"). InnerVolumeSpecName "kube-api-access-phjr2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:01:42.053607 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:42.053564    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19bae71e-c48b-4b51-9e5c-ea7ce7735057-maas-api-xks-ca" pod "19bae71e-c48b-4b51-9e5c-ea7ce7735057" (UID: "19bae71e-c48b-4b51-9e5c-ea7ce7735057"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:01:42.151204 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:42.151161    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfps8\" (UniqueName: \"kubernetes.io/projected/dc6e70c9-ff92-4d6b-bf03-28441c0750c3-kube-api-access-dfps8\") pod \"maas-api-revoke-sub-19cf1ad9-gkr8q\" (UID: \"dc6e70c9-ff92-4d6b-bf03-28441c0750c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19cf1ad9-gkr8q"
Oct 30 21:01:42.151421 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:42.151292    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc6e70c9-ff92-4d6b-bf03-28441c0750c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-19cf1ad9-gkr8q\" (UID: \"dc6e70c9-ff92-4d6b-bf03-28441c0750c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19cf1ad9-gkr8q"
Oct 30 21:01:42.151421 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:42.151360    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc6e70c9-ff92-4d6b-bf03-28441c0750c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-19cf1ad9-gkr8q\" (UID: \"dc6e70c9-ff92-4d6b-bf03-28441c0750c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19cf1ad9-gkr8q"
Oct 30 21:01:42.151552 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:42.151428    4521 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19bae71e-c48b-4b51-9e5c-ea7ce7735057-maas-api-openshift-ca\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:01:42.151552 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:42.151446    4521 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19bae71e-c48b-4b51-9e5c-ea7ce7735057-maas-api-xks-ca\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:01:42.151552 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:42.151460    4521 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phjr2\" (UniqueName: \"kubernetes.io/projected/19bae71e-c48b-4b51-9e5c-ea7ce7735057-kube-api-access-phjr2\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:01:42.152106 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:42.152079    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc6e70c9-ff92-4d6b-bf03-28441c0750c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-19cf1ad9-gkr8q\" (UID: \"dc6e70c9-ff92-4d6b-bf03-28441c0750c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19cf1ad9-gkr8q"
Oct 30 21:01:42.153762 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:42.153739    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc6e70c9-ff92-4d6b-bf03-28441c0750c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-19cf1ad9-gkr8q\" (UID: \"dc6e70c9-ff92-4d6b-bf03-28441c0750c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19cf1ad9-gkr8q"
Oct 30 21:01:42.163572 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:42.163548    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfps8\" (UniqueName: \"kubernetes.io/projected/dc6e70c9-ff92-4d6b-bf03-28441c0750c3-kube-api-access-dfps8\") pod \"maas-api-revoke-sub-19cf1ad9-gkr8q\" (UID: \"dc6e70c9-ff92-4d6b-bf03-28441c0750c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19cf1ad9-gkr8q"
Oct 30 21:01:42.356048 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:42.355944    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19cf1ad9-gkr8q"
Oct 30 21:01:42.678244 ip-10-0-131-125 kubenswrapper[4521]: W1030 21:01:42.678209    4521 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc6e70c9_ff92_4d6b_bf03_28441c0750c3.slice/crio-dee10c0ea3aa5c7b27639da3e9e0c3e642d74682bd577512bbdef2aeba32fab1 WatchSource:0}: Error finding container dee10c0ea3aa5c7b27639da3e9e0c3e642d74682bd577512bbdef2aeba32fab1: Status 404 returned error can't find the container with id dee10c0ea3aa5c7b27639da3e9e0c3e642d74682bd577512bbdef2aeba32fab1
Oct 30 21:01:42.681173 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:42.681146    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19cf1ad9-gkr8q"]
Oct 30 21:01:42.766612 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:42.766575    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19cf1ad9-gkr8q" event={"ID":"dc6e70c9-ff92-4d6b-bf03-28441c0750c3","Type":"ContainerStarted","Data":"dee10c0ea3aa5c7b27639da3e9e0c3e642d74682bd577512bbdef2aeba32fab1"}
Oct 30 21:01:42.769385 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:42.768090    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bad53a71-964rp"
Oct 30 21:01:42.769385 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:42.768125    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bad53a71-964rp" event={"ID":"19bae71e-c48b-4b51-9e5c-ea7ce7735057","Type":"ContainerDied","Data":"448dff48db294560e26702a976066bc247c76c589548fb6921c43e0a1b8836f6"}
Oct 30 21:01:42.769385 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:42.768210    4521 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="448dff48db294560e26702a976066bc247c76c589548fb6921c43e0a1b8836f6"
Oct 30 21:01:43.779726 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:43.779691    4521 generic.go:356] "Generic (PLEG): container finished" podID="dc6e70c9-ff92-4d6b-bf03-28441c0750c3" containerID="9fcfadf36e0ec120d8224f77357065db91c3d84d6fb9ea5c65d608fdfef742b6" exitCode=0
Oct 30 21:01:43.780154 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:43.779779    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19cf1ad9-gkr8q" event={"ID":"dc6e70c9-ff92-4d6b-bf03-28441c0750c3","Type":"ContainerDied","Data":"9fcfadf36e0ec120d8224f77357065db91c3d84d6fb9ea5c65d608fdfef742b6"}
Oct 30 21:01:44.963121 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:44.963089    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19cf1ad9-gkr8q"
Oct 30 21:01:45.091494 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:45.091410    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc6e70c9-ff92-4d6b-bf03-28441c0750c3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc6e70c9-ff92-4d6b-bf03-28441c0750c3-maas-api-openshift-ca\") pod \"dc6e70c9-ff92-4d6b-bf03-28441c0750c3\" (UID: \"dc6e70c9-ff92-4d6b-bf03-28441c0750c3\") "
Oct 30 21:01:45.091647 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:45.091616    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc6e70c9-ff92-4d6b-bf03-28441c0750c3-kube-api-access-dfps8\" (UniqueName: \"kubernetes.io/projected/dc6e70c9-ff92-4d6b-bf03-28441c0750c3-kube-api-access-dfps8\") pod \"dc6e70c9-ff92-4d6b-bf03-28441c0750c3\" (UID: \"dc6e70c9-ff92-4d6b-bf03-28441c0750c3\") "
Oct 30 21:01:45.091711 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:45.091699    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc6e70c9-ff92-4d6b-bf03-28441c0750c3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc6e70c9-ff92-4d6b-bf03-28441c0750c3-maas-api-xks-ca\") pod \"dc6e70c9-ff92-4d6b-bf03-28441c0750c3\" (UID: \"dc6e70c9-ff92-4d6b-bf03-28441c0750c3\") "
Oct 30 21:01:45.091792 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:45.091769    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc6e70c9-ff92-4d6b-bf03-28441c0750c3-maas-api-openshift-ca" pod "dc6e70c9-ff92-4d6b-bf03-28441c0750c3" (UID: "dc6e70c9-ff92-4d6b-bf03-28441c0750c3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:01:45.092097 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:45.092081    4521 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc6e70c9-ff92-4d6b-bf03-28441c0750c3-maas-api-openshift-ca\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:01:45.093846 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:45.093819    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc6e70c9-ff92-4d6b-bf03-28441c0750c3-maas-api-xks-ca" pod "dc6e70c9-ff92-4d6b-bf03-28441c0750c3" (UID: "dc6e70c9-ff92-4d6b-bf03-28441c0750c3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:01:45.093942 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:45.093862    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc6e70c9-ff92-4d6b-bf03-28441c0750c3-kube-api-access-dfps8" pod "dc6e70c9-ff92-4d6b-bf03-28441c0750c3" (UID: "dc6e70c9-ff92-4d6b-bf03-28441c0750c3"). InnerVolumeSpecName "kube-api-access-dfps8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:01:45.193905 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:45.193863    4521 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc6e70c9-ff92-4d6b-bf03-28441c0750c3-maas-api-xks-ca\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:01:45.193905 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:45.193903    4521 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dfps8\" (UniqueName: \"kubernetes.io/projected/dc6e70c9-ff92-4d6b-bf03-28441c0750c3-kube-api-access-dfps8\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:01:45.796889 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:45.796850    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19cf1ad9-gkr8q" event={"ID":"dc6e70c9-ff92-4d6b-bf03-28441c0750c3","Type":"ContainerDied","Data":"dee10c0ea3aa5c7b27639da3e9e0c3e642d74682bd577512bbdef2aeba32fab1"}
Oct 30 21:01:45.796889 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:45.796878    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19cf1ad9-gkr8q"
Oct 30 21:01:45.797116 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:45.796904    4521 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dee10c0ea3aa5c7b27639da3e9e0c3e642d74682bd577512bbdef2aeba32fab1"
Oct 30 21:01:46.441508 ip-10-0-131-125 kubenswrapper[4521]: E1030 21:01:46.441469    4521 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lkn2k" containerName="extractor"
Oct 30 21:01:47.359665 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:47.359621    4521 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-65968b/distinct-2-w5-65968b-kserve-6d65c85f6c-dz5r7"
Oct 30 21:01:47.378526 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:47.378489    4521 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-65968b/distinct-2-w5-65968b-kserve-6d65c85f6c-dz5r7"
Oct 30 21:01:47.826862 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:47.826828    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-65968b/distinct-2-w5-65968b-kserve-6d65c85f6c-dz5r7"
Oct 30 21:01:47.899660 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:47.899625    4521 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-39fb96/premium-w6-39fb96-kserve-b9cf7f75c-96blx"]
Oct 30 21:01:53.252533 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:53.252492    4521 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-39fb96-7954dbb4bc-r8z6k"]
Oct 30 21:01:54.708560 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:54.708518    4521 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-32a969-49xn7"]
Oct 30 21:01:54.709189 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:54.709169    4521 cpu_manager.go:414] "containerMap: removing container" podUID="dc6e70c9-ff92-4d6b-bf03-28441c0750c3" containerName="revoke-keys"
Oct 30 21:01:54.709189 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:54.709190    4521 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc6e70c9-ff92-4d6b-bf03-28441c0750c3" containerName="revoke-keys"
Oct 30 21:01:54.709298 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:54.709289    4521 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc6e70c9-ff92-4d6b-bf03-28441c0750c3" containerName="revoke-keys"
Oct 30 21:01:54.714430 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:54.714400    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-32a969-49xn7"
Oct 30 21:01:54.726707 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:54.726676    4521 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c45pw\""
Oct 30 21:01:54.727848 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:54.727827    4521 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:01:54.737854 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:54.737824    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-32a969-49xn7"]
Oct 30 21:01:54.744828 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:54.744802    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxj6q\" (UniqueName: \"kubernetes.io/projected/080ab3d3-8e29-4214-9b65-b9a2ce6f0f3a-kube-api-access-fxj6q\") pod \"maas-api-revoke-keys-e2e-worker-w2-32a969-49xn7\" (UID: \"080ab3d3-8e29-4214-9b65-b9a2ce6f0f3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-32a969-49xn7"
Oct 30 21:01:54.745108 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:54.745088    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/080ab3d3-8e29-4214-9b65-b9a2ce6f0f3a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-32a969-49xn7\" (UID: \"080ab3d3-8e29-4214-9b65-b9a2ce6f0f3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-32a969-49xn7"
Oct 30 21:01:54.745413 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:54.745396    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/080ab3d3-8e29-4214-9b65-b9a2ce6f0f3a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-32a969-49xn7\" (UID: \"080ab3d3-8e29-4214-9b65-b9a2ce6f0f3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-32a969-49xn7"
Oct 30 21:01:54.846939 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:54.846902    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/080ab3d3-8e29-4214-9b65-b9a2ce6f0f3a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-32a969-49xn7\" (UID: \"080ab3d3-8e29-4214-9b65-b9a2ce6f0f3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-32a969-49xn7"
Oct 30 21:01:54.847497 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:54.847006    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxj6q\" (UniqueName: \"kubernetes.io/projected/080ab3d3-8e29-4214-9b65-b9a2ce6f0f3a-kube-api-access-fxj6q\") pod \"maas-api-revoke-keys-e2e-worker-w2-32a969-49xn7\" (UID: \"080ab3d3-8e29-4214-9b65-b9a2ce6f0f3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-32a969-49xn7"
Oct 30 21:01:54.847497 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:54.847044    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/080ab3d3-8e29-4214-9b65-b9a2ce6f0f3a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-32a969-49xn7\" (UID: \"080ab3d3-8e29-4214-9b65-b9a2ce6f0f3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-32a969-49xn7"
Oct 30 21:01:54.847788 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:54.847764    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/080ab3d3-8e29-4214-9b65-b9a2ce6f0f3a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-32a969-49xn7\" (UID: \"080ab3d3-8e29-4214-9b65-b9a2ce6f0f3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-32a969-49xn7"
Oct 30 21:01:54.849548 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:54.849524    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/080ab3d3-8e29-4214-9b65-b9a2ce6f0f3a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-32a969-49xn7\" (UID: \"080ab3d3-8e29-4214-9b65-b9a2ce6f0f3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-32a969-49xn7"
Oct 30 21:01:54.860349 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:54.860301    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxj6q\" (UniqueName: \"kubernetes.io/projected/080ab3d3-8e29-4214-9b65-b9a2ce6f0f3a-kube-api-access-fxj6q\") pod \"maas-api-revoke-keys-e2e-worker-w2-32a969-49xn7\" (UID: \"080ab3d3-8e29-4214-9b65-b9a2ce6f0f3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-32a969-49xn7"
Oct 30 21:01:55.047262 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:55.047167    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-32a969-49xn7"
Oct 30 21:01:55.331417 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:55.331386    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-32a969-49xn7"]
Oct 30 21:01:55.333502 ip-10-0-131-125 kubenswrapper[4521]: W1030 21:01:55.333472    4521 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod080ab3d3_8e29_4214_9b65_b9a2ce6f0f3a.slice/crio-9ece821cee8bb71be1e560900ab7150960a1e60013dc0d1a4fa559b5eab62be7 WatchSource:0}: Error finding container 9ece821cee8bb71be1e560900ab7150960a1e60013dc0d1a4fa559b5eab62be7: Status 404 returned error can't find the container with id 9ece821cee8bb71be1e560900ab7150960a1e60013dc0d1a4fa559b5eab62be7
Oct 30 21:01:55.581433 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:55.581317    4521 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-5tqcb"]
Oct 30 21:01:55.586665 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:55.586631    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-5tqcb"
Oct 30 21:01:55.601982 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:55.601950    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-5tqcb"]
Oct 30 21:01:55.658970 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:55.658728    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11b85d0f-fa82-4861-9c1c-e47a1c1c308a-istiod-ca-cert\") pod \"e2e-worker-w0-4a0c5b-openshift-default-676865df99-5tqcb\" (UID: \"11b85d0f-fa82-4861-9c1c-e47a1c1c308a\") " pod="openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-5tqcb"
Oct 30 21:01:55.658970 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:55.658880    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11b85d0f-fa82-4861-9c1c-e47a1c1c308a-istio-envoy\") pod \"e2e-worker-w0-4a0c5b-openshift-default-676865df99-5tqcb\" (UID: \"11b85d0f-fa82-4861-9c1c-e47a1c1c308a\") " pod="openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-5tqcb"
Oct 30 21:01:55.658970 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:55.658931    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11b85d0f-fa82-4861-9c1c-e47a1c1c308a-istio-data\") pod \"e2e-worker-w0-4a0c5b-openshift-default-676865df99-5tqcb\" (UID: \"11b85d0f-fa82-4861-9c1c-e47a1c1c308a\") " pod="openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-5tqcb"
Oct 30 21:01:55.659296 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:55.659101    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11b85d0f-fa82-4861-9c1c-e47a1c1c308a-workload-certs\") pod \"e2e-worker-w0-4a0c5b-openshift-default-676865df99-5tqcb\" (UID: \"11b85d0f-fa82-4861-9c1c-e47a1c1c308a\") " pod="openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-5tqcb"
Oct 30 21:01:55.659296 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:55.659136    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11b85d0f-fa82-4861-9c1c-e47a1c1c308a-workload-socket\") pod \"e2e-worker-w0-4a0c5b-openshift-default-676865df99-5tqcb\" (UID: \"11b85d0f-fa82-4861-9c1c-e47a1c1c308a\") " pod="openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-5tqcb"
Oct 30 21:01:55.659296 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:55.659177    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11b85d0f-fa82-4861-9c1c-e47a1c1c308a-credential-socket\") pod \"e2e-worker-w0-4a0c5b-openshift-default-676865df99-5tqcb\" (UID: \"11b85d0f-fa82-4861-9c1c-e47a1c1c308a\") " pod="openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-5tqcb"
Oct 30 21:01:55.659296 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:55.659221    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11b85d0f-fa82-4861-9c1c-e47a1c1c308a-istio-token\") pod \"e2e-worker-w0-4a0c5b-openshift-default-676865df99-5tqcb\" (UID: \"11b85d0f-fa82-4861-9c1c-e47a1c1c308a\") " pod="openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-5tqcb"
Oct 30 21:01:55.659568 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:55.659368    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11b85d0f-fa82-4861-9c1c-e47a1c1c308a-istio-podinfo\") pod \"e2e-worker-w0-4a0c5b-openshift-default-676865df99-5tqcb\" (UID: \"11b85d0f-fa82-4861-9c1c-e47a1c1c308a\") " pod="openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-5tqcb"
Oct 30 21:01:55.659568 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:55.659400    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6lrz\" (UniqueName: \"kubernetes.io/projected/11b85d0f-fa82-4861-9c1c-e47a1c1c308a-kube-api-access-m6lrz\") pod \"e2e-worker-w0-4a0c5b-openshift-default-676865df99-5tqcb\" (UID: \"11b85d0f-fa82-4861-9c1c-e47a1c1c308a\") " pod="openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-5tqcb"
Oct 30 21:01:55.764380 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:55.760830    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11b85d0f-fa82-4861-9c1c-e47a1c1c308a-istio-podinfo\") pod \"e2e-worker-w0-4a0c5b-openshift-default-676865df99-5tqcb\" (UID: \"11b85d0f-fa82-4861-9c1c-e47a1c1c308a\") " pod="openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-5tqcb"
Oct 30 21:01:55.764380 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:55.760886    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6lrz\" (UniqueName: \"kubernetes.io/projected/11b85d0f-fa82-4861-9c1c-e47a1c1c308a-kube-api-access-m6lrz\") pod \"e2e-worker-w0-4a0c5b-openshift-default-676865df99-5tqcb\" (UID: \"11b85d0f-fa82-4861-9c1c-e47a1c1c308a\") " pod="openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-5tqcb"
Oct 30 21:01:55.764380 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:55.760926    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11b85d0f-fa82-4861-9c1c-e47a1c1c308a-istiod-ca-cert\") pod \"e2e-worker-w0-4a0c5b-openshift-default-676865df99-5tqcb\" (UID: \"11b85d0f-fa82-4861-9c1c-e47a1c1c308a\") " pod="openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-5tqcb"
Oct 30 21:01:55.764380 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:55.760963    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11b85d0f-fa82-4861-9c1c-e47a1c1c308a-istio-envoy\") pod \"e2e-worker-w0-4a0c5b-openshift-default-676865df99-5tqcb\" (UID: \"11b85d0f-fa82-4861-9c1c-e47a1c1c308a\") " pod="openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-5tqcb"
Oct 30 21:01:55.764380 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:55.760992    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11b85d0f-fa82-4861-9c1c-e47a1c1c308a-istio-data\") pod \"e2e-worker-w0-4a0c5b-openshift-default-676865df99-5tqcb\" (UID: \"11b85d0f-fa82-4861-9c1c-e47a1c1c308a\") " pod="openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-5tqcb"
Oct 30 21:01:55.764380 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:55.761140    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11b85d0f-fa82-4861-9c1c-e47a1c1c308a-workload-certs\") pod \"e2e-worker-w0-4a0c5b-openshift-default-676865df99-5tqcb\" (UID: \"11b85d0f-fa82-4861-9c1c-e47a1c1c308a\") " pod="openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-5tqcb"
Oct 30 21:01:55.764380 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:55.761175    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11b85d0f-fa82-4861-9c1c-e47a1c1c308a-workload-socket\") pod \"e2e-worker-w0-4a0c5b-openshift-default-676865df99-5tqcb\" (UID: \"11b85d0f-fa82-4861-9c1c-e47a1c1c308a\") " pod="openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-5tqcb"
Oct 30 21:01:55.764380 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:55.761213    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11b85d0f-fa82-4861-9c1c-e47a1c1c308a-credential-socket\") pod \"e2e-worker-w0-4a0c5b-openshift-default-676865df99-5tqcb\" (UID: \"11b85d0f-fa82-4861-9c1c-e47a1c1c308a\") " pod="openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-5tqcb"
Oct 30 21:01:55.764380 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:55.761254    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11b85d0f-fa82-4861-9c1c-e47a1c1c308a-istio-token\") pod \"e2e-worker-w0-4a0c5b-openshift-default-676865df99-5tqcb\" (UID: \"11b85d0f-fa82-4861-9c1c-e47a1c1c308a\") " pod="openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-5tqcb"
Oct 30 21:01:55.764380 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:55.762533    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11b85d0f-fa82-4861-9c1c-e47a1c1c308a-istiod-ca-cert\") pod \"e2e-worker-w0-4a0c5b-openshift-default-676865df99-5tqcb\" (UID: \"11b85d0f-fa82-4861-9c1c-e47a1c1c308a\") " pod="openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-5tqcb"
Oct 30 21:01:55.765371 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:55.764480    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11b85d0f-fa82-4861-9c1c-e47a1c1c308a-istio-podinfo\") pod \"e2e-worker-w0-4a0c5b-openshift-default-676865df99-5tqcb\" (UID: \"11b85d0f-fa82-4861-9c1c-e47a1c1c308a\") " pod="openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-5tqcb"
Oct 30 21:01:55.765371 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:55.764751    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11b85d0f-fa82-4861-9c1c-e47a1c1c308a-workload-certs\") pod \"e2e-worker-w0-4a0c5b-openshift-default-676865df99-5tqcb\" (UID: \"11b85d0f-fa82-4861-9c1c-e47a1c1c308a\") " pod="openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-5tqcb"
Oct 30 21:01:55.765371 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:55.765004    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11b85d0f-fa82-4861-9c1c-e47a1c1c308a-istio-data\") pod \"e2e-worker-w0-4a0c5b-openshift-default-676865df99-5tqcb\" (UID: \"11b85d0f-fa82-4861-9c1c-e47a1c1c308a\") " pod="openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-5tqcb"
Oct 30 21:01:55.765371 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:55.765266    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11b85d0f-fa82-4861-9c1c-e47a1c1c308a-workload-socket\") pod \"e2e-worker-w0-4a0c5b-openshift-default-676865df99-5tqcb\" (UID: \"11b85d0f-fa82-4861-9c1c-e47a1c1c308a\") " pod="openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-5tqcb"
Oct 30 21:01:55.765670 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:55.765492    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11b85d0f-fa82-4861-9c1c-e47a1c1c308a-istio-envoy\") pod \"e2e-worker-w0-4a0c5b-openshift-default-676865df99-5tqcb\" (UID: \"11b85d0f-fa82-4861-9c1c-e47a1c1c308a\") " pod="openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-5tqcb"
Oct 30 21:01:55.765670 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:55.765550    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11b85d0f-fa82-4861-9c1c-e47a1c1c308a-credential-socket\") pod \"e2e-worker-w0-4a0c5b-openshift-default-676865df99-5tqcb\" (UID: \"11b85d0f-fa82-4861-9c1c-e47a1c1c308a\") " pod="openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-5tqcb"
Oct 30 21:01:55.774944 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:55.774916    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6lrz\" (UniqueName: \"kubernetes.io/projected/11b85d0f-fa82-4861-9c1c-e47a1c1c308a-kube-api-access-m6lrz\") pod \"e2e-worker-w0-4a0c5b-openshift-default-676865df99-5tqcb\" (UID: \"11b85d0f-fa82-4861-9c1c-e47a1c1c308a\") " pod="openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-5tqcb"
Oct 30 21:01:55.774944 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:55.774929    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11b85d0f-fa82-4861-9c1c-e47a1c1c308a-istio-token\") pod \"e2e-worker-w0-4a0c5b-openshift-default-676865df99-5tqcb\" (UID: \"11b85d0f-fa82-4861-9c1c-e47a1c1c308a\") " pod="openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-5tqcb"
Oct 30 21:01:55.886885 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:55.886793    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-32a969-49xn7" event={"ID":"080ab3d3-8e29-4214-9b65-b9a2ce6f0f3a","Type":"ContainerStarted","Data":"d789c2cd2837495b6fc2814d9d24c7e5369b4d7973a3018bbe7519b278dd793b"}
Oct 30 21:01:55.886885 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:55.886861    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-32a969-49xn7" event={"ID":"080ab3d3-8e29-4214-9b65-b9a2ce6f0f3a","Type":"ContainerStarted","Data":"9ece821cee8bb71be1e560900ab7150960a1e60013dc0d1a4fa559b5eab62be7"}
Oct 30 21:01:55.904944 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:55.904917    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-5tqcb"
Oct 30 21:01:55.936791 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:55.936722    4521 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-32a969-49xn7" podStartSLOduration=1.9367011669999998 podStartE2EDuration="1.936701167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:01:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:01:55.934142831 +0000 UTC m=+1383.279764069" watchObservedRunningTime="2026-10-30 21:01:55.936701167 +0000 UTC m=+1383.282322468"
Oct 30 21:01:56.228726 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:56.217742    4521 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:01:56.228726 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:56.217908    4521 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:01:56.228726 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:56.218004    4521 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:01:56.228726 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:56.225543    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-5tqcb"]
Oct 30 21:01:56.896320 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:56.896286    4521 generic.go:356] "Generic (PLEG): container finished" podID="080ab3d3-8e29-4214-9b65-b9a2ce6f0f3a" containerID="d789c2cd2837495b6fc2814d9d24c7e5369b4d7973a3018bbe7519b278dd793b" exitCode=0
Oct 30 21:01:56.896320 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:56.896362    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-32a969-49xn7" event={"ID":"080ab3d3-8e29-4214-9b65-b9a2ce6f0f3a","Type":"ContainerDied","Data":"d789c2cd2837495b6fc2814d9d24c7e5369b4d7973a3018bbe7519b278dd793b"}
Oct 30 21:01:56.898505 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:56.898480    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-5tqcb" event={"ID":"11b85d0f-fa82-4861-9c1c-e47a1c1c308a","Type":"ContainerStarted","Data":"b251af91f4b3645d981cdae3ca14049d5f598b6d3c79722fa36f70e80e806ab9"}
Oct 30 21:01:56.898636 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:56.898528    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-5tqcb" event={"ID":"11b85d0f-fa82-4861-9c1c-e47a1c1c308a","Type":"ContainerStarted","Data":"e44287839751a0524eb32b14aa1c652e3797b084f0a5720a9de8ff4c38451f9f"}
Oct 30 21:01:56.905392 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:56.905358    4521 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-5tqcb"
Oct 30 21:01:56.948542 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:56.948479    4521 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-5tqcb" podStartSLOduration=1.948459614 podStartE2EDuration="1.948459614s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:01:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:01:56.939133884 +0000 UTC m=+1384.284755123" watchObservedRunningTime="2026-10-30 21:01:56.948459614 +0000 UTC m=+1384.294080853"
Oct 30 21:01:57.906834 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:57.906805    4521 patch_prober.go:29] interesting pod/e2e-worker-w0-4a0c5b-openshift-default-676865df99-5tqcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 30 21:01:57.907189 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:57.906855    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-5tqcb" podUID="11b85d0f-fa82-4861-9c1c-e47a1c1c308a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 30 21:01:58.061362 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:58.060070    4521 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-261c359b-nxwd6"]
Oct 30 21:01:58.065894 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:58.065841    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-261c359b-nxwd6"]
Oct 30 21:01:58.066038 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:58.065945    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-261c359b-nxwd6"
Oct 30 21:01:58.093952 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:58.093916    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/233fc24c-d8c6-4aab-9d22-a68b05a9b378-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-261c359b-nxwd6\" (UID: \"233fc24c-d8c6-4aab-9d22-a68b05a9b378\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-261c359b-nxwd6"
Oct 30 21:01:58.094109 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:58.094069    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psclz\" (UniqueName: \"kubernetes.io/projected/233fc24c-d8c6-4aab-9d22-a68b05a9b378-kube-api-access-psclz\") pod \"maas-api-revoke-sub-261c359b-nxwd6\" (UID: \"233fc24c-d8c6-4aab-9d22-a68b05a9b378\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-261c359b-nxwd6"
Oct 30 21:01:58.094183 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:58.094107    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/233fc24c-d8c6-4aab-9d22-a68b05a9b378-maas-api-xks-ca\") pod \"maas-api-revoke-sub-261c359b-nxwd6\" (UID: \"233fc24c-d8c6-4aab-9d22-a68b05a9b378\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-261c359b-nxwd6"
Oct 30 21:01:58.199388 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:58.195509    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/233fc24c-d8c6-4aab-9d22-a68b05a9b378-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-261c359b-nxwd6\" (UID: \"233fc24c-d8c6-4aab-9d22-a68b05a9b378\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-261c359b-nxwd6"
Oct 30 21:01:58.199388 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:58.195695    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psclz\" (UniqueName: \"kubernetes.io/projected/233fc24c-d8c6-4aab-9d22-a68b05a9b378-kube-api-access-psclz\") pod \"maas-api-revoke-sub-261c359b-nxwd6\" (UID: \"233fc24c-d8c6-4aab-9d22-a68b05a9b378\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-261c359b-nxwd6"
Oct 30 21:01:58.199388 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:58.195733    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/233fc24c-d8c6-4aab-9d22-a68b05a9b378-maas-api-xks-ca\") pod \"maas-api-revoke-sub-261c359b-nxwd6\" (UID: \"233fc24c-d8c6-4aab-9d22-a68b05a9b378\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-261c359b-nxwd6"
Oct 30 21:01:58.212684 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:58.210665    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/233fc24c-d8c6-4aab-9d22-a68b05a9b378-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-261c359b-nxwd6\" (UID: \"233fc24c-d8c6-4aab-9d22-a68b05a9b378\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-261c359b-nxwd6"
Oct 30 21:01:58.241343 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:58.237732    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/233fc24c-d8c6-4aab-9d22-a68b05a9b378-maas-api-xks-ca\") pod \"maas-api-revoke-sub-261c359b-nxwd6\" (UID: \"233fc24c-d8c6-4aab-9d22-a68b05a9b378\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-261c359b-nxwd6"
Oct 30 21:01:58.264352 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:58.257942    4521 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-39fb96-7954dbb4bc-r8z6k" podUID="f3530279-92bb-445f-95b2-441c22fadc9f" containerName="payload-pre-processing" containerID="cri-o://57592b57782433ff180994a4f8417e3267dac98c3d770d97e6aad2722facb942" gracePeriod=25
Oct 30 21:01:58.283400 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:58.273680    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-psclz\" (UniqueName: \"kubernetes.io/projected/233fc24c-d8c6-4aab-9d22-a68b05a9b378-kube-api-access-psclz\") pod \"maas-api-revoke-sub-261c359b-nxwd6\" (UID: \"233fc24c-d8c6-4aab-9d22-a68b05a9b378\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-261c359b-nxwd6"
Oct 30 21:01:58.283400 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:58.275203    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-32a969-49xn7"
Oct 30 21:01:58.391756 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:58.390569    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-261c359b-nxwd6"
Oct 30 21:01:58.446889 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:58.446842    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/080ab3d3-8e29-4214-9b65-b9a2ce6f0f3a-kube-api-access-fxj6q\" (UniqueName: \"kubernetes.io/projected/080ab3d3-8e29-4214-9b65-b9a2ce6f0f3a-kube-api-access-fxj6q\") pod \"080ab3d3-8e29-4214-9b65-b9a2ce6f0f3a\" (UID: \"080ab3d3-8e29-4214-9b65-b9a2ce6f0f3a\") "
Oct 30 21:01:58.447676 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:58.447653    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/080ab3d3-8e29-4214-9b65-b9a2ce6f0f3a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/080ab3d3-8e29-4214-9b65-b9a2ce6f0f3a-maas-api-xks-ca\") pod \"080ab3d3-8e29-4214-9b65-b9a2ce6f0f3a\" (UID: \"080ab3d3-8e29-4214-9b65-b9a2ce6f0f3a\") "
Oct 30 21:01:58.448054 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:58.448026    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/080ab3d3-8e29-4214-9b65-b9a2ce6f0f3a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/080ab3d3-8e29-4214-9b65-b9a2ce6f0f3a-maas-api-openshift-ca\") pod \"080ab3d3-8e29-4214-9b65-b9a2ce6f0f3a\" (UID: \"080ab3d3-8e29-4214-9b65-b9a2ce6f0f3a\") "
Oct 30 21:01:58.450778 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:58.450707    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/080ab3d3-8e29-4214-9b65-b9a2ce6f0f3a-maas-api-openshift-ca" pod "080ab3d3-8e29-4214-9b65-b9a2ce6f0f3a" (UID: "080ab3d3-8e29-4214-9b65-b9a2ce6f0f3a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:01:58.504343 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:58.500905    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/080ab3d3-8e29-4214-9b65-b9a2ce6f0f3a-maas-api-xks-ca" pod "080ab3d3-8e29-4214-9b65-b9a2ce6f0f3a" (UID: "080ab3d3-8e29-4214-9b65-b9a2ce6f0f3a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:01:58.507823 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:58.506269    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/080ab3d3-8e29-4214-9b65-b9a2ce6f0f3a-kube-api-access-fxj6q" pod "080ab3d3-8e29-4214-9b65-b9a2ce6f0f3a" (UID: "080ab3d3-8e29-4214-9b65-b9a2ce6f0f3a"). InnerVolumeSpecName "kube-api-access-fxj6q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:01:58.552345 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:58.551541    4521 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/080ab3d3-8e29-4214-9b65-b9a2ce6f0f3a-maas-api-openshift-ca\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:01:58.552345 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:58.551586    4521 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fxj6q\" (UniqueName: \"kubernetes.io/projected/080ab3d3-8e29-4214-9b65-b9a2ce6f0f3a-kube-api-access-fxj6q\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:01:58.552345 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:58.551604    4521 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/080ab3d3-8e29-4214-9b65-b9a2ce6f0f3a-maas-api-xks-ca\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:01:58.783454 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:58.768744    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-39fb96-7954dbb4bc-r8z6k" podUID="f3530279-92bb-445f-95b2-441c22fadc9f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:01:58.907350 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:58.907171    4521 patch_prober.go:29] interesting pod/e2e-worker-w0-4a0c5b-openshift-default-676865df99-5tqcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 30 21:01:58.907350 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:58.907231    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-5tqcb" podUID="11b85d0f-fa82-4861-9c1c-e47a1c1c308a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 30 21:01:59.549493 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:01:59.549449    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-261c359b-nxwd6"]
Oct 30 21:01:59.637349 ip-10-0-131-125 kubenswrapper[4521]: W1030 21:01:59.614820    4521 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod233fc24c_d8c6_4aab_9d22_a68b05a9b378.slice/crio-17385a31f2f9179506183cc61e7b99d047f74449aaf62e926d121d79c6af5cff WatchSource:0}: Error finding container 17385a31f2f9179506183cc61e7b99d047f74449aaf62e926d121d79c6af5cff: Status 404 returned error can't find the container with id 17385a31f2f9179506183cc61e7b99d047f74449aaf62e926d121d79c6af5cff
Oct 30 21:02:00.001488 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:00.001443    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a8f94/simulator-w3-5a8f94-kserve-77cc88cc89-n4sl6" podUID="1fa6900c-eee2-4f00-b0ec-248def48dfc4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:02:00.090386 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:00.090359    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-32a969-49xn7"
Oct 30 21:02:00.090889 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:00.090866    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-32a969-49xn7" event={"ID":"080ab3d3-8e29-4214-9b65-b9a2ce6f0f3a","Type":"ContainerDied","Data":"9ece821cee8bb71be1e560900ab7150960a1e60013dc0d1a4fa559b5eab62be7"}
Oct 30 21:02:00.091017 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:00.091006    4521 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9ece821cee8bb71be1e560900ab7150960a1e60013dc0d1a4fa559b5eab62be7"
Oct 30 21:02:00.098102 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:00.098064    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-32a969/distinct-w2-32a969-kserve-7f798b664f-jdh82" podUID="887f1d27-bf24-4b57-b6e4-8fb095df7a50" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded"
Oct 30 21:02:00.102741 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:00.098286    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c4b379/premium-w1-c4b379-kserve-575f896d49-2n6gz" podUID="2aac3e8e-d631-4b9c-bdbb-3b4b1df133a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:02:00.130351 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:00.104317    4521 patch_prober.go:29] interesting pod/e2e-worker-w0-4a0c5b-openshift-default-676865df99-5tqcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 30 21:02:00.130715 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:00.130681    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-5tqcb" podUID="11b85d0f-fa82-4861-9c1c-e47a1c1c308a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 30 21:02:00.172344 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:00.137444    4521 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-32a969/distinct-2-w2-32a969-kserve-5c7845448b-vt5gw"]
Oct 30 21:02:00.180408 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:00.179427    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c2d1eb/premium-w4-c2d1eb-kserve-8696fbd9b9-mxqhm" podUID="4bb0f898-b77f-419d-bff4-6413e82ef51d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:02:00.180408 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:00.179836    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a8f94/distinct-w3-5a8f94-kserve-5f8b544d59-mf95j" podUID="b8ac1cca-4ebd-474c-b32f-a3595a4cd847" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded"
Oct 30 21:02:00.201532 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:00.199978    4521 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-32a969/distinct-w2-32a969-kserve-7f798b664f-jdh82"]
Oct 30 21:02:00.243121 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:00.236750    4521 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-32a969/simulator-w2-32a969-kserve-6cc8dfc44c-dwhlt"]
Oct 30 21:02:00.344975 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:00.343724    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-32a969/distinct-2-w2-32a969-kserve-5c7845448b-vt5gw" podUID="d430ed01-65d1-43a6-a0c4-ab82212376a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:02:00.494347 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:00.493656    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-32a969/simulator-w2-32a969-kserve-6cc8dfc44c-dwhlt" podUID="62a6adf7-d17a-4fec-a707-a2d549b67753" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:02:00.498343 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:00.496400    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-39fb96/premium-w6-39fb96-kserve-b9cf7f75c-96blx" podUID="18ca0a94-c322-41fa-9fd9-0f14419b5075" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:02:00.550619 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:00.549446    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c4b379/distinct-2-w1-c4b379-kserve-7999b785f5-97fz5" podUID="080a654d-a5bf-4845-a07f-cc8a7e6d0ace" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:02:00.550995 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:00.550896    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a8f94/premium-w3-5a8f94-kserve-bb96b6848-5hk8h" podUID="24ccc37d-027a-4097-8da5-04df417aef94" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Oct 30 21:02:00.642579 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:00.641593    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a8f94/distinct-2-w3-5a8f94-kserve-ffcc6449b-hv48k" podUID="bc0f5352-5c44-4c35-adf5-3fffab06b644" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:02:00.831347 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:00.830903    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-65968b/premium-w5-65968b-kserve-67977485f7-c9jpv" podUID="92b2dd65-8a83-436d-afa8-50e5f0d11fa0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 30 21:02:00.876109 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:00.876065    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-65968b/distinct-2-w5-65968b-kserve-6d65c85f6c-dz5r7" podUID="d6751cf5-7df8-4ffa-b6c8-8b0c4f16ba2a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:02:00.895407 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:00.876677    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a0c5b/premium-w0-4a0c5b-kserve-85dcf89f57-mscdn" podUID="00e42758-77ce-4663-adfa-2202c0aae24a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:02:00.906523 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:00.905630    4521 patch_prober.go:29] interesting pod/e2e-worker-w0-4a0c5b-openshift-default-676865df99-5tqcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 30 21:02:00.906523 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:00.905686    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-5tqcb" podUID="11b85d0f-fa82-4861-9c1c-e47a1c1c308a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 30 21:02:01.006357 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:01.005447    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a8f94/simulator-w3-5a8f94-kserve-77cc88cc89-n4sl6" podUID="1fa6900c-eee2-4f00-b0ec-248def48dfc4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 30 21:02:01.006357 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:01.006235    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5a8f94/simulator-w3-5a8f94-kserve-77cc88cc89-n4sl6"
Oct 30 21:02:01.103556 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:01.103507    4521 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-32a969/distinct-2-w2-32a969-kserve-5c7845448b-vt5gw" podUID="d430ed01-65d1-43a6-a0c4-ab82212376a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:02:01.104366 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:01.104319    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c4b379/premium-w1-c4b379-kserve-575f896d49-2n6gz" podUID="2aac3e8e-d631-4b9c-bdbb-3b4b1df133a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 30 21:02:01.200357 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:01.134620    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-32a969/distinct-w2-32a969-kserve-7f798b664f-jdh82" podUID="887f1d27-bf24-4b57-b6e4-8fb095df7a50" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:02:01.213489 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:01.200971    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c4b379/premium-w1-c4b379-kserve-575f896d49-2n6gz"
Oct 30 21:02:01.213489 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:01.201035    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-32a969/distinct-w2-32a969-kserve-7f798b664f-jdh82"
Oct 30 21:02:01.213489 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:01.197893    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c2d1eb/premium-w4-c2d1eb-kserve-8696fbd9b9-mxqhm" podUID="4bb0f898-b77f-419d-bff4-6413e82ef51d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded"
Oct 30 21:02:01.213489 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:01.202373    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c2d1eb/premium-w4-c2d1eb-kserve-8696fbd9b9-mxqhm"
Oct 30 21:02:01.244301 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:01.197849    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a8f94/distinct-w3-5a8f94-kserve-5f8b544d59-mf95j" podUID="b8ac1cca-4ebd-474c-b32f-a3595a4cd847" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:02:01.244301 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:01.242965    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5a8f94/distinct-w3-5a8f94-kserve-5f8b544d59-mf95j"
Oct 30 21:02:01.362981 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:01.362935    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-32a969/distinct-2-w2-32a969-kserve-5c7845448b-vt5gw" podUID="d430ed01-65d1-43a6-a0c4-ab82212376a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:02:01.388358 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:01.371823    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-32a969/distinct-2-w2-32a969-kserve-5c7845448b-vt5gw"
Oct 30 21:02:01.476585 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:01.460993    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-261c359b-nxwd6" event={"ID":"233fc24c-d8c6-4aab-9d22-a68b05a9b378","Type":"ContainerStarted","Data":"e80142b954d75601b661e585265be56c91785e7ddf5825513b4e605e0e09c3d8"}
Oct 30 21:02:01.476585 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:01.461103    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-261c359b-nxwd6" event={"ID":"233fc24c-d8c6-4aab-9d22-a68b05a9b378","Type":"ContainerStarted","Data":"17385a31f2f9179506183cc61e7b99d047f74449aaf62e926d121d79c6af5cff"}
Oct 30 21:02:01.496692 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:01.496647    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-39fb96/premium-w6-39fb96-kserve-b9cf7f75c-96blx" podUID="18ca0a94-c322-41fa-9fd9-0f14419b5075" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded"
Oct 30 21:02:01.531359 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:01.507339    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-39fb96/premium-w6-39fb96-kserve-b9cf7f75c-96blx"
Oct 30 21:02:01.531628 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:01.509740    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-32a969/simulator-w2-32a969-kserve-6cc8dfc44c-dwhlt" podUID="62a6adf7-d17a-4fec-a707-a2d549b67753" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:02:01.546377 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:01.541517    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-32a969/simulator-w2-32a969-kserve-6cc8dfc44c-dwhlt"
Oct 30 21:02:01.546377 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:01.567571    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a8f94/premium-w3-5a8f94-kserve-bb96b6848-5hk8h" podUID="24ccc37d-027a-4097-8da5-04df417aef94" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:02:01.546377 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:01.568358    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c4b379/distinct-2-w1-c4b379-kserve-7999b785f5-97fz5" podUID="080a654d-a5bf-4845-a07f-cc8a7e6d0ace" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": context deadline exceeded"
Oct 30 21:02:01.546377 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:01.568884    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5a8f94/premium-w3-5a8f94-kserve-bb96b6848-5hk8h"
Oct 30 21:02:01.546377 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:01.568941    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c4b379/distinct-2-w1-c4b379-kserve-7999b785f5-97fz5"
Oct 30 21:02:01.666559 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:01.648370    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a8f94/distinct-2-w3-5a8f94-kserve-ffcc6449b-hv48k" podUID="bc0f5352-5c44-4c35-adf5-3fffab06b644" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:02:01.666559 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:01.651295    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5a8f94/distinct-2-w3-5a8f94-kserve-ffcc6449b-hv48k"
Oct 30 21:02:01.835924 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:01.835806    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-65968b/premium-w5-65968b-kserve-67977485f7-c9jpv" podUID="92b2dd65-8a83-436d-afa8-50e5f0d11fa0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 30 21:02:01.837865 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:01.837832    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-65968b/premium-w5-65968b-kserve-67977485f7-c9jpv"
Oct 30 21:02:01.900419 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:01.877409    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-65968b/distinct-2-w5-65968b-kserve-6d65c85f6c-dz5r7" podUID="d6751cf5-7df8-4ffa-b6c8-8b0c4f16ba2a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": context deadline exceeded"
Oct 30 21:02:01.900419 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:01.878873    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-65968b/distinct-2-w5-65968b-kserve-6d65c85f6c-dz5r7"
Oct 30 21:02:01.924379 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:01.903210    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a0c5b/premium-w0-4a0c5b-kserve-85dcf89f57-mscdn" podUID="00e42758-77ce-4663-adfa-2202c0aae24a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:02:01.924379 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:01.903748    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4a0c5b/premium-w0-4a0c5b-kserve-85dcf89f57-mscdn"
Oct 30 21:02:01.941127 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:01.935117    4521 patch_prober.go:29] interesting pod/e2e-worker-w0-4a0c5b-openshift-default-676865df99-5tqcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 30 21:02:01.941791 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:01.941754    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-5tqcb" podUID="11b85d0f-fa82-4861-9c1c-e47a1c1c308a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 30 21:02:02.028435 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:02.028390    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a8f94/simulator-w3-5a8f94-kserve-77cc88cc89-n4sl6" podUID="1fa6900c-eee2-4f00-b0ec-248def48dfc4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 30 21:02:02.081804 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:02.081738    4521 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-5a8f94/premium-w3-5a8f94-kserve-bb96b6848-5hk8h" podUID="24ccc37d-027a-4097-8da5-04df417aef94" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:02:02.194425 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:02.174718    4521 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-261c359b-nxwd6" podStartSLOduration=5.174695215 podStartE2EDuration="5.174695215s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:01:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:02:01.604356172 +0000 UTC m=+1388.949977412" watchObservedRunningTime="2026-10-30 21:02:02.174695215 +0000 UTC m=+1389.520316456"
Oct 30 21:02:02.218526 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:02.216880    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-32a969/distinct-w2-32a969-kserve-7f798b664f-jdh82" podUID="887f1d27-bf24-4b57-b6e4-8fb095df7a50" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded"
Oct 30 21:02:02.218526 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:02.217402    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c4b379/premium-w1-c4b379-kserve-575f896d49-2n6gz" podUID="2aac3e8e-d631-4b9c-bdbb-3b4b1df133a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 30 21:02:02.220829 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:02.220796    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c2d1eb/premium-w4-c2d1eb-kserve-8696fbd9b9-mxqhm" podUID="4bb0f898-b77f-419d-bff4-6413e82ef51d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:02:02.260257 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:02.260200    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a8f94/distinct-w3-5a8f94-kserve-5f8b544d59-mf95j" podUID="b8ac1cca-4ebd-474c-b32f-a3595a4cd847" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:02:02.364472 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:02.364424    4521 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-c4b379/premium-w1-c4b379-kserve-575f896d49-2n6gz" podUID="2aac3e8e-d631-4b9c-bdbb-3b4b1df133a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:02:02.388774 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:02.388022    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-32a969/distinct-2-w2-32a969-kserve-5c7845448b-vt5gw" podUID="d430ed01-65d1-43a6-a0c4-ab82212376a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:02:02.525410 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:02.509422    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-39fb96/premium-w6-39fb96-kserve-b9cf7f75c-96blx" podUID="18ca0a94-c322-41fa-9fd9-0f14419b5075" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:02:02.545222 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:02.533890    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-32a969/simulator-w2-32a969-kserve-6cc8dfc44c-dwhlt" podUID="62a6adf7-d17a-4fec-a707-a2d549b67753" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:02:02.587350 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:02.570504    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c4b379/distinct-2-w1-c4b379-kserve-7999b785f5-97fz5" podUID="080a654d-a5bf-4845-a07f-cc8a7e6d0ace" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:02:02.587350 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:02.570791    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a8f94/premium-w3-5a8f94-kserve-bb96b6848-5hk8h" podUID="24ccc37d-027a-4097-8da5-04df417aef94" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Oct 30 21:02:02.642037 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:02.641996    4521 generic.go:356] "Generic (PLEG): container finished" podID="233fc24c-d8c6-4aab-9d22-a68b05a9b378" containerID="e80142b954d75601b661e585265be56c91785e7ddf5825513b4e605e0e09c3d8" exitCode=0
Oct 30 21:02:02.642544 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:02.642513    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-261c359b-nxwd6" event={"ID":"233fc24c-d8c6-4aab-9d22-a68b05a9b378","Type":"ContainerDied","Data":"e80142b954d75601b661e585265be56c91785e7ddf5825513b4e605e0e09c3d8"}
Oct 30 21:02:02.662094 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:02.662051    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a8f94/distinct-2-w3-5a8f94-kserve-ffcc6449b-hv48k" podUID="bc0f5352-5c44-4c35-adf5-3fffab06b644" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:02:02.714045 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:02.714012    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-32a969/simulator-w2-32a969-kserve-6cc8dfc44c-dwhlt"
Oct 30 21:02:02.864441 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:02.848684    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-65968b/premium-w5-65968b-kserve-67977485f7-c9jpv" podUID="92b2dd65-8a83-436d-afa8-50e5f0d11fa0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 30 21:02:02.909605 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:02.878114    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-65968b/distinct-2-w5-65968b-kserve-6d65c85f6c-dz5r7" podUID="d6751cf5-7df8-4ffa-b6c8-8b0c4f16ba2a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:02:02.992350 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:02.910734    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a0c5b/premium-w0-4a0c5b-kserve-85dcf89f57-mscdn" podUID="00e42758-77ce-4663-adfa-2202c0aae24a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:02:03.055488 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:03.055438    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a8f94/simulator-w3-5a8f94-kserve-77cc88cc89-n4sl6" podUID="1fa6900c-eee2-4f00-b0ec-248def48dfc4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 30 21:02:03.082081 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:03.082042    4521 patch_prober.go:29] interesting pod/e2e-worker-w0-4a0c5b-openshift-default-676865df99-5tqcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 30 21:02:03.104352 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:03.104290    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-5tqcb" podUID="11b85d0f-fa82-4861-9c1c-e47a1c1c308a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 30 21:02:03.106513 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:03.106411    4521 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-4a0c5b/premium-w0-4a0c5b-kserve-85dcf89f57-mscdn" podUID="00e42758-77ce-4663-adfa-2202c0aae24a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:02:03.114299 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:03.114262    4521 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-39fb96/premium-w6-39fb96-kserve-b9cf7f75c-96blx" podUID="18ca0a94-c322-41fa-9fd9-0f14419b5075" containerName="main" containerID="cri-o://e70b88f888be048aa9ffe8bbc08b0dcc8c03d1e8c89116f6df4e0a302aeb05ff" gracePeriod=45
Oct 30 21:02:03.224360 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:03.223450    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c4b379/premium-w1-c4b379-kserve-575f896d49-2n6gz" podUID="2aac3e8e-d631-4b9c-bdbb-3b4b1df133a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 30 21:02:03.224360 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:03.223882    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-32a969/distinct-w2-32a969-kserve-7f798b664f-jdh82" podUID="887f1d27-bf24-4b57-b6e4-8fb095df7a50" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded"
Oct 30 21:02:03.225002 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:03.224968    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c2d1eb/premium-w4-c2d1eb-kserve-8696fbd9b9-mxqhm" podUID="4bb0f898-b77f-419d-bff4-6413e82ef51d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:02:03.262454 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:03.262410    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a8f94/distinct-w3-5a8f94-kserve-5f8b544d59-mf95j" podUID="b8ac1cca-4ebd-474c-b32f-a3595a4cd847" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded"
Oct 30 21:02:03.396594 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:03.396476    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-32a969/distinct-2-w2-32a969-kserve-5c7845448b-vt5gw" podUID="d430ed01-65d1-43a6-a0c4-ab82212376a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:02:03.515609 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:03.515548    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-39fb96/premium-w6-39fb96-kserve-b9cf7f75c-96blx" podUID="18ca0a94-c322-41fa-9fd9-0f14419b5075" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:02:03.589609 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:03.589565    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c4b379/distinct-2-w1-c4b379-kserve-7999b785f5-97fz5" podUID="080a654d-a5bf-4845-a07f-cc8a7e6d0ace" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": context deadline exceeded"
Oct 30 21:02:03.622356 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:03.606188    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5a8f94/simulator-w3-5a8f94-kserve-77cc88cc89-n4sl6"
Oct 30 21:02:03.622356 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:03.606955    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a8f94/premium-w3-5a8f94-kserve-bb96b6848-5hk8h" podUID="24ccc37d-027a-4097-8da5-04df417aef94" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:02:03.663592 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:03.662456    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a8f94/distinct-2-w3-5a8f94-kserve-ffcc6449b-hv48k" podUID="bc0f5352-5c44-4c35-adf5-3fffab06b644" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:02:03.886351 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:03.884742    4521 patch_prober.go:29] interesting pod/e2e-worker-w0-4a0c5b-openshift-default-676865df99-hlv6g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:02:03.886351 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:03.884806    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-hlv6g" podUID="f860c372-4493-4c25-b46a-a6d3af710390" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:02:03.947407 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:03.888007    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-65968b/premium-w5-65968b-kserve-67977485f7-c9jpv" podUID="92b2dd65-8a83-436d-afa8-50e5f0d11fa0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 30 21:02:03.949835 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:03.937344    4521 patch_prober.go:29] interesting pod/e2e-worker-w0-4a0c5b-openshift-default-676865df99-5tqcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 30 21:02:03.951175 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:03.951143    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-5tqcb" podUID="11b85d0f-fa82-4861-9c1c-e47a1c1c308a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 30 21:02:04.077453 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:04.077405    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-65968b/distinct-2-w5-65968b-kserve-6d65c85f6c-dz5r7" podUID="d6751cf5-7df8-4ffa-b6c8-8b0c4f16ba2a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:02:04.079165 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:04.079130    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a0c5b/premium-w0-4a0c5b-kserve-85dcf89f57-mscdn" podUID="00e42758-77ce-4663-adfa-2202c0aae24a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:02:04.232403 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:04.228915    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c4b379/premium-w1-c4b379-kserve-575f896d49-2n6gz" podUID="2aac3e8e-d631-4b9c-bdbb-3b4b1df133a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 30 21:02:04.232403 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:04.229490    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-32a969/distinct-w2-32a969-kserve-7f798b664f-jdh82" podUID="887f1d27-bf24-4b57-b6e4-8fb095df7a50" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded"
Oct 30 21:02:04.232403 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:04.231735    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c2d1eb/premium-w4-c2d1eb-kserve-8696fbd9b9-mxqhm" podUID="4bb0f898-b77f-419d-bff4-6413e82ef51d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded"
Oct 30 21:02:04.292366 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:04.274816    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a8f94/distinct-w3-5a8f94-kserve-5f8b544d59-mf95j" podUID="b8ac1cca-4ebd-474c-b32f-a3595a4cd847" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:02:04.408060 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:04.407735    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-32a969/distinct-2-w2-32a969-kserve-5c7845448b-vt5gw" podUID="d430ed01-65d1-43a6-a0c4-ab82212376a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:02:04.475178 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:04.475139    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-32a969/distinct-2-w2-32a969-kserve-5c7845448b-vt5gw"
Oct 30 21:02:04.496604 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:04.492480    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-32a969/simulator-w2-32a969-kserve-6cc8dfc44c-dwhlt" podUID="62a6adf7-d17a-4fec-a707-a2d549b67753" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:02:04.602551 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:04.602506    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-39fb96/premium-w6-39fb96-kserve-b9cf7f75c-96blx" podUID="18ca0a94-c322-41fa-9fd9-0f14419b5075" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded"
Oct 30 21:02:04.627883 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:04.627667    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a8f94/premium-w3-5a8f94-kserve-bb96b6848-5hk8h" podUID="24ccc37d-027a-4097-8da5-04df417aef94" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:02:04.628516 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:04.628090    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c4b379/distinct-2-w1-c4b379-kserve-7999b785f5-97fz5" podUID="080a654d-a5bf-4845-a07f-cc8a7e6d0ace" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:02:04.685649 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:04.676432    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a8f94/distinct-2-w3-5a8f94-kserve-ffcc6449b-hv48k" podUID="bc0f5352-5c44-4c35-adf5-3fffab06b644" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": context deadline exceeded"
Oct 30 21:02:04.901530 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:04.900124    4521 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-5a8f94/distinct-2-w3-5a8f94-kserve-ffcc6449b-hv48k" podUID="bc0f5352-5c44-4c35-adf5-3fffab06b644" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:02:04.907217 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:04.906775    4521 patch_prober.go:29] interesting pod/e2e-worker-w0-4a0c5b-openshift-default-676865df99-5tqcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 30 21:02:04.910531 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:04.910490    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-5tqcb" podUID="11b85d0f-fa82-4861-9c1c-e47a1c1c308a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 30 21:02:04.978144 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:04.978080    4521 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-c4b379/distinct-2-w1-c4b379-kserve-7999b785f5-97fz5" podUID="080a654d-a5bf-4845-a07f-cc8a7e6d0ace" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/health\": context deadline exceeded"
Oct 30 21:02:04.978375 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:04.978205    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-65968b/premium-w5-65968b-kserve-67977485f7-c9jpv" podUID="92b2dd65-8a83-436d-afa8-50e5f0d11fa0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 30 21:02:05.088351 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:05.076480    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a8f94/simulator-w3-5a8f94-kserve-77cc88cc89-n4sl6" podUID="1fa6900c-eee2-4f00-b0ec-248def48dfc4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:02:05.088351 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:05.080434    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-65968b/distinct-2-w5-65968b-kserve-6d65c85f6c-dz5r7" podUID="d6751cf5-7df8-4ffa-b6c8-8b0c4f16ba2a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:02:05.105368 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:05.102538    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a0c5b/premium-w0-4a0c5b-kserve-85dcf89f57-mscdn" podUID="00e42758-77ce-4663-adfa-2202c0aae24a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:02:05.131535 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:05.131476    4521 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-5a8f94/distinct-w3-5a8f94-kserve-5f8b544d59-mf95j" podUID="b8ac1cca-4ebd-474c-b32f-a3595a4cd847" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:02:05.162694 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:05.162600    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-65968b/premium-w5-65968b-kserve-67977485f7-c9jpv"
Oct 30 21:02:05.162973 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:05.162957    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5a8f94/premium-w3-5a8f94-kserve-bb96b6848-5hk8h"
Oct 30 21:02:05.244903 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:05.242569    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c4b379/premium-w1-c4b379-kserve-575f896d49-2n6gz" podUID="2aac3e8e-d631-4b9c-bdbb-3b4b1df133a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 30 21:02:05.244903 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:05.243872    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c2d1eb/premium-w4-c2d1eb-kserve-8696fbd9b9-mxqhm" podUID="4bb0f898-b77f-419d-bff4-6413e82ef51d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded"
Oct 30 21:02:05.276010 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:05.275978    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a8f94/distinct-w3-5a8f94-kserve-5f8b544d59-mf95j" podUID="b8ac1cca-4ebd-474c-b32f-a3595a4cd847" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded"
Oct 30 21:02:05.288757 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:05.288690    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-32a969/distinct-w2-32a969-kserve-7f798b664f-jdh82" podUID="887f1d27-bf24-4b57-b6e4-8fb095df7a50" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:02:05.409883 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:05.409851    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-261c359b-nxwd6"
Oct 30 21:02:05.511004 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:05.509556    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-32a969/simulator-w2-32a969-kserve-6cc8dfc44c-dwhlt" podUID="62a6adf7-d17a-4fec-a707-a2d549b67753" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:02:05.511004 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:05.510298    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-32a969/simulator-w2-32a969-kserve-6cc8dfc44c-dwhlt"
Oct 30 21:02:05.511938 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:05.511910    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/233fc24c-d8c6-4aab-9d22-a68b05a9b378-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/233fc24c-d8c6-4aab-9d22-a68b05a9b378-maas-api-openshift-ca\") pod \"233fc24c-d8c6-4aab-9d22-a68b05a9b378\" (UID: \"233fc24c-d8c6-4aab-9d22-a68b05a9b378\") "
Oct 30 21:02:05.540015 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:05.539985    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/233fc24c-d8c6-4aab-9d22-a68b05a9b378-kube-api-access-psclz\" (UniqueName: \"kubernetes.io/projected/233fc24c-d8c6-4aab-9d22-a68b05a9b378-kube-api-access-psclz\") pod \"233fc24c-d8c6-4aab-9d22-a68b05a9b378\" (UID: \"233fc24c-d8c6-4aab-9d22-a68b05a9b378\") "
Oct 30 21:02:05.562672 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:05.562641    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/233fc24c-d8c6-4aab-9d22-a68b05a9b378-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/233fc24c-d8c6-4aab-9d22-a68b05a9b378-maas-api-xks-ca\") pod \"233fc24c-d8c6-4aab-9d22-a68b05a9b378\" (UID: \"233fc24c-d8c6-4aab-9d22-a68b05a9b378\") "
Oct 30 21:02:05.618967 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:05.605111    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-39fb96/premium-w6-39fb96-kserve-b9cf7f75c-96blx" podUID="18ca0a94-c322-41fa-9fd9-0f14419b5075" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:02:05.620460 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:05.521561    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/233fc24c-d8c6-4aab-9d22-a68b05a9b378-maas-api-openshift-ca" pod "233fc24c-d8c6-4aab-9d22-a68b05a9b378" (UID: "233fc24c-d8c6-4aab-9d22-a68b05a9b378"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:02:05.620699 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:05.620674    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-39fb96/premium-w6-39fb96-kserve-b9cf7f75c-96blx" podUID="18ca0a94-c322-41fa-9fd9-0f14419b5075" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": dial tcp 10.133.0.57:8000: connect: connection refused"
Oct 30 21:02:05.685277 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:05.685228    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a8f94/distinct-2-w3-5a8f94-kserve-ffcc6449b-hv48k" podUID="bc0f5352-5c44-4c35-adf5-3fffab06b644" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:02:05.686782 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:05.686750    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c4b379/distinct-2-w1-c4b379-kserve-7999b785f5-97fz5" podUID="080a654d-a5bf-4845-a07f-cc8a7e6d0ace" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:02:05.725346 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:05.723608    4521 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/233fc24c-d8c6-4aab-9d22-a68b05a9b378-maas-api-openshift-ca\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:05.881163 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:05.881077    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/233fc24c-d8c6-4aab-9d22-a68b05a9b378-maas-api-xks-ca" pod "233fc24c-d8c6-4aab-9d22-a68b05a9b378" (UID: "233fc24c-d8c6-4aab-9d22-a68b05a9b378"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:02:05.965637 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:05.965604    4521 patch_prober.go:29] interesting pod/e2e-worker-w0-4a0c5b-openshift-default-676865df99-5tqcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 30 21:02:06.060363 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:06.043554    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-5tqcb" podUID="11b85d0f-fa82-4861-9c1c-e47a1c1c308a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 30 21:02:06.060363 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:05.967337    4521 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/233fc24c-d8c6-4aab-9d22-a68b05a9b378-maas-api-xks-ca\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:06.060363 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:05.966041    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-65968b/distinct-2-w5-65968b-kserve-6d65c85f6c-dz5r7"
Oct 30 21:02:06.060363 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:06.056516    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4a0c5b/premium-w0-4a0c5b-kserve-85dcf89f57-mscdn"
Oct 30 21:02:06.272549 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:06.272501    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c4b379/premium-w1-c4b379-kserve-575f896d49-2n6gz" podUID="2aac3e8e-d631-4b9c-bdbb-3b4b1df133a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 30 21:02:06.286678 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:06.286613    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a8f94/distinct-w3-5a8f94-kserve-5f8b544d59-mf95j" podUID="b8ac1cca-4ebd-474c-b32f-a3595a4cd847" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:02:06.335750 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:06.287022    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/233fc24c-d8c6-4aab-9d22-a68b05a9b378-kube-api-access-psclz" pod "233fc24c-d8c6-4aab-9d22-a68b05a9b378" (UID: "233fc24c-d8c6-4aab-9d22-a68b05a9b378"). InnerVolumeSpecName "kube-api-access-psclz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:02:06.336438 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:06.296434    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c2d1eb/premium-w4-c2d1eb-kserve-8696fbd9b9-mxqhm" podUID="4bb0f898-b77f-419d-bff4-6413e82ef51d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:02:06.370026 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:06.369983    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c4b379/premium-w1-c4b379-kserve-575f896d49-2n6gz"
Oct 30 21:02:06.403112 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:06.403085    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c4b379/distinct-2-w1-c4b379-kserve-7999b785f5-97fz5"
Oct 30 21:02:06.448732 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:06.446825    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5a8f94/distinct-2-w3-5a8f94-kserve-ffcc6449b-hv48k"
Oct 30 21:02:06.450176 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:06.450145    4521 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-psclz\" (UniqueName: \"kubernetes.io/projected/233fc24c-d8c6-4aab-9d22-a68b05a9b378-kube-api-access-psclz\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:06.450606 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:06.450566    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-32a969/distinct-w2-32a969-kserve-7f798b664f-jdh82" podUID="887f1d27-bf24-4b57-b6e4-8fb095df7a50" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:02:06.475092 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:06.475065    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5a8f94/distinct-w3-5a8f94-kserve-5f8b544d59-mf95j"
Oct 30 21:02:06.477990 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:06.477936    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-39fb96/premium-w6-39fb96-kserve-b9cf7f75c-96blx" podUID="18ca0a94-c322-41fa-9fd9-0f14419b5075" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": dial tcp 10.133.0.57:8000: connect: connection refused"
Oct 30 21:02:06.478254 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:06.478238    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-32a969/simulator-w2-32a969-kserve-6cc8dfc44c-dwhlt"
Oct 30 21:02:06.481557 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:06.481475    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c2d1eb/premium-w4-c2d1eb-kserve-8696fbd9b9-mxqhm"
Oct 30 21:02:06.482098 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:06.482070    4521 generic.go:356] "Generic (PLEG): container finished" podID="18ca0a94-c322-41fa-9fd9-0f14419b5075" containerID="e70b88f888be048aa9ffe8bbc08b0dcc8c03d1e8c89116f6df4e0a302aeb05ff" exitCode=0
Oct 30 21:02:06.482395 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:06.482375    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-39fb96/premium-w6-39fb96-kserve-b9cf7f75c-96blx" event={"ID":"18ca0a94-c322-41fa-9fd9-0f14419b5075","Type":"ContainerDied","Data":"e70b88f888be048aa9ffe8bbc08b0dcc8c03d1e8c89116f6df4e0a302aeb05ff"}
Oct 30 21:02:06.484707 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:06.484687    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-261c359b-nxwd6" event={"ID":"233fc24c-d8c6-4aab-9d22-a68b05a9b378","Type":"ContainerDied","Data":"17385a31f2f9179506183cc61e7b99d047f74449aaf62e926d121d79c6af5cff"}
Oct 30 21:02:06.484846 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:06.484833    4521 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="17385a31f2f9179506183cc61e7b99d047f74449aaf62e926d121d79c6af5cff"
Oct 30 21:02:06.484982 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:06.484972    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-261c359b-nxwd6"
Oct 30 21:02:06.494141 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:06.494109    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-32a969/distinct-w2-32a969-kserve-7f798b664f-jdh82"
Oct 30 21:02:06.832598 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:06.832560    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-39fb96/premium-w6-39fb96-kserve-b9cf7f75c-96blx"
Oct 30 21:02:06.906224 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:06.906190    4521 patch_prober.go:29] interesting pod/e2e-worker-w0-4a0c5b-openshift-default-676865df99-5tqcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 30 21:02:06.906424 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:06.906242    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-5tqcb" podUID="11b85d0f-fa82-4861-9c1c-e47a1c1c308a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 30 21:02:06.973689 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:06.973649    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18ca0a94-c322-41fa-9fd9-0f14419b5075-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18ca0a94-c322-41fa-9fd9-0f14419b5075-tmp-dir\") pod \"18ca0a94-c322-41fa-9fd9-0f14419b5075\" (UID: \"18ca0a94-c322-41fa-9fd9-0f14419b5075\") "
Oct 30 21:02:06.973856 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:06.973727    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18ca0a94-c322-41fa-9fd9-0f14419b5075-dshm\" (UniqueName: \"kubernetes.io/empty-dir/18ca0a94-c322-41fa-9fd9-0f14419b5075-dshm\") pod \"18ca0a94-c322-41fa-9fd9-0f14419b5075\" (UID: \"18ca0a94-c322-41fa-9fd9-0f14419b5075\") "
Oct 30 21:02:06.973856 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:06.973832    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18ca0a94-c322-41fa-9fd9-0f14419b5075-tls-certs\" (UniqueName: \"kubernetes.io/secret/18ca0a94-c322-41fa-9fd9-0f14419b5075-tls-certs\") pod \"18ca0a94-c322-41fa-9fd9-0f14419b5075\" (UID: \"18ca0a94-c322-41fa-9fd9-0f14419b5075\") "
Oct 30 21:02:06.973988 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:06.973965    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18ca0a94-c322-41fa-9fd9-0f14419b5075-kube-api-access-j4bbk\" (UniqueName: \"kubernetes.io/projected/18ca0a94-c322-41fa-9fd9-0f14419b5075-kube-api-access-j4bbk\") pod \"18ca0a94-c322-41fa-9fd9-0f14419b5075\" (UID: \"18ca0a94-c322-41fa-9fd9-0f14419b5075\") "
Oct 30 21:02:06.974118 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:06.974022    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18ca0a94-c322-41fa-9fd9-0f14419b5075-home\" (UniqueName: \"kubernetes.io/empty-dir/18ca0a94-c322-41fa-9fd9-0f14419b5075-home\") pod \"18ca0a94-c322-41fa-9fd9-0f14419b5075\" (UID: \"18ca0a94-c322-41fa-9fd9-0f14419b5075\") "
Oct 30 21:02:06.974181 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:06.974126    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18ca0a94-c322-41fa-9fd9-0f14419b5075-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18ca0a94-c322-41fa-9fd9-0f14419b5075-model-cache\") pod \"18ca0a94-c322-41fa-9fd9-0f14419b5075\" (UID: \"18ca0a94-c322-41fa-9fd9-0f14419b5075\") "
Oct 30 21:02:06.974386 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:06.974243    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18ca0a94-c322-41fa-9fd9-0f14419b5075-tmp-dir" pod "18ca0a94-c322-41fa-9fd9-0f14419b5075" (UID: "18ca0a94-c322-41fa-9fd9-0f14419b5075"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:02:06.974480 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:06.974464    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18ca0a94-c322-41fa-9fd9-0f14419b5075-home" pod "18ca0a94-c322-41fa-9fd9-0f14419b5075" (UID: "18ca0a94-c322-41fa-9fd9-0f14419b5075"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:02:06.974552 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:06.974518    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18ca0a94-c322-41fa-9fd9-0f14419b5075-model-cache" pod "18ca0a94-c322-41fa-9fd9-0f14419b5075" (UID: "18ca0a94-c322-41fa-9fd9-0f14419b5075"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:02:06.974847 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:06.974828    4521 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18ca0a94-c322-41fa-9fd9-0f14419b5075-home\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:06.974938 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:06.974849    4521 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18ca0a94-c322-41fa-9fd9-0f14419b5075-model-cache\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:06.974938 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:06.974858    4521 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18ca0a94-c322-41fa-9fd9-0f14419b5075-tmp-dir\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:06.976481 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:06.976456    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18ca0a94-c322-41fa-9fd9-0f14419b5075-dshm" pod "18ca0a94-c322-41fa-9fd9-0f14419b5075" (UID: "18ca0a94-c322-41fa-9fd9-0f14419b5075"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:02:06.976589 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:06.976571    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18ca0a94-c322-41fa-9fd9-0f14419b5075-kube-api-access-j4bbk" pod "18ca0a94-c322-41fa-9fd9-0f14419b5075" (UID: "18ca0a94-c322-41fa-9fd9-0f14419b5075"). InnerVolumeSpecName "kube-api-access-j4bbk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:02:06.976804 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:06.976783    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18ca0a94-c322-41fa-9fd9-0f14419b5075-tls-certs" pod "18ca0a94-c322-41fa-9fd9-0f14419b5075" (UID: "18ca0a94-c322-41fa-9fd9-0f14419b5075"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:02:07.076312 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:07.076219    4521 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4bbk\" (UniqueName: \"kubernetes.io/projected/18ca0a94-c322-41fa-9fd9-0f14419b5075-kube-api-access-j4bbk\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:07.076312 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:07.076258    4521 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18ca0a94-c322-41fa-9fd9-0f14419b5075-dshm\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:07.076312 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:07.076277    4521 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18ca0a94-c322-41fa-9fd9-0f14419b5075-tls-certs\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:07.494498 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:07.494462    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-39fb96/premium-w6-39fb96-kserve-b9cf7f75c-96blx"
Oct 30 21:02:07.494960 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:07.494469    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-39fb96/premium-w6-39fb96-kserve-b9cf7f75c-96blx" event={"ID":"18ca0a94-c322-41fa-9fd9-0f14419b5075","Type":"ContainerDied","Data":"6071fcf4e071d65a907c727159a694ae100b0ce9c4482a0f4d9c467fe33920ba"}
Oct 30 21:02:07.494960 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:07.494581    4521 scope.go:122] "RemoveContainer" containerID="e70b88f888be048aa9ffe8bbc08b0dcc8c03d1e8c89116f6df4e0a302aeb05ff"
Oct 30 21:02:07.525014 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:07.524985    4521 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-39fb96/premium-w6-39fb96-kserve-b9cf7f75c-96blx"]
Oct 30 21:02:07.532248 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:07.532215    4521 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-39fb96/premium-w6-39fb96-kserve-b9cf7f75c-96blx"]
Oct 30 21:02:07.770159 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:07.770071    4521 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63ecfd7f-vjzz8"]
Oct 30 21:02:07.770741 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:07.770720    4521 cpu_manager.go:414] "containerMap: removing container" podUID="18ca0a94-c322-41fa-9fd9-0f14419b5075" containerName="main"
Oct 30 21:02:07.770861 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:07.770742    4521 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18ca0a94-c322-41fa-9fd9-0f14419b5075" containerName="main"
Oct 30 21:02:07.770861 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:07.770781    4521 cpu_manager.go:414] "containerMap: removing container" podUID="233fc24c-d8c6-4aab-9d22-a68b05a9b378" containerName="revoke-keys"
Oct 30 21:02:07.770861 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:07.770789    4521 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="233fc24c-d8c6-4aab-9d22-a68b05a9b378" containerName="revoke-keys"
Oct 30 21:02:07.770861 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:07.770806    4521 cpu_manager.go:414] "containerMap: removing container" podUID="080ab3d3-8e29-4214-9b65-b9a2ce6f0f3a" containerName="revoke-keys"
Oct 30 21:02:07.770861 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:07.770814    4521 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="080ab3d3-8e29-4214-9b65-b9a2ce6f0f3a" containerName="revoke-keys"
Oct 30 21:02:07.771103 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:07.770984    4521 memory_manager.go:365] "RemoveStaleState removing state" podUID="18ca0a94-c322-41fa-9fd9-0f14419b5075" containerName="main"
Oct 30 21:02:07.771103 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:07.771000    4521 memory_manager.go:365] "RemoveStaleState removing state" podUID="233fc24c-d8c6-4aab-9d22-a68b05a9b378" containerName="revoke-keys"
Oct 30 21:02:07.771103 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:07.771011    4521 memory_manager.go:365] "RemoveStaleState removing state" podUID="080ab3d3-8e29-4214-9b65-b9a2ce6f0f3a" containerName="revoke-keys"
Oct 30 21:02:07.776221 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:07.776199    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63ecfd7f-vjzz8"
Oct 30 21:02:07.782807 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:07.782614    4521 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:02:07.784627 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:07.783200    4521 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c45pw\""
Oct 30 21:02:07.784627 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:07.783897    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63ecfd7f-vjzz8"]
Oct 30 21:02:07.890024 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:07.889981    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc97df13-49df-4ee8-bcde-b775f4f594cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-63ecfd7f-vjzz8\" (UID: \"fc97df13-49df-4ee8-bcde-b775f4f594cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63ecfd7f-vjzz8"
Oct 30 21:02:07.890219 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:07.890194    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5gwg\" (UniqueName: \"kubernetes.io/projected/fc97df13-49df-4ee8-bcde-b775f4f594cd-kube-api-access-l5gwg\") pod \"maas-api-revoke-sub-63ecfd7f-vjzz8\" (UID: \"fc97df13-49df-4ee8-bcde-b775f4f594cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63ecfd7f-vjzz8"
Oct 30 21:02:07.890312 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:07.890255    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc97df13-49df-4ee8-bcde-b775f4f594cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-63ecfd7f-vjzz8\" (UID: \"fc97df13-49df-4ee8-bcde-b775f4f594cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63ecfd7f-vjzz8"
Oct 30 21:02:07.991831 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:07.991790    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5gwg\" (UniqueName: \"kubernetes.io/projected/fc97df13-49df-4ee8-bcde-b775f4f594cd-kube-api-access-l5gwg\") pod \"maas-api-revoke-sub-63ecfd7f-vjzz8\" (UID: \"fc97df13-49df-4ee8-bcde-b775f4f594cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63ecfd7f-vjzz8"
Oct 30 21:02:07.992030 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:07.991895    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc97df13-49df-4ee8-bcde-b775f4f594cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-63ecfd7f-vjzz8\" (UID: \"fc97df13-49df-4ee8-bcde-b775f4f594cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63ecfd7f-vjzz8"
Oct 30 21:02:07.992030 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:07.992001    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc97df13-49df-4ee8-bcde-b775f4f594cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-63ecfd7f-vjzz8\" (UID: \"fc97df13-49df-4ee8-bcde-b775f4f594cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63ecfd7f-vjzz8"
Oct 30 21:02:07.993025 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:07.992993    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc97df13-49df-4ee8-bcde-b775f4f594cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-63ecfd7f-vjzz8\" (UID: \"fc97df13-49df-4ee8-bcde-b775f4f594cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63ecfd7f-vjzz8"
Oct 30 21:02:07.995598 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:07.995574    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc97df13-49df-4ee8-bcde-b775f4f594cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-63ecfd7f-vjzz8\" (UID: \"fc97df13-49df-4ee8-bcde-b775f4f594cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63ecfd7f-vjzz8"
Oct 30 21:02:08.005608 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:08.005570    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5gwg\" (UniqueName: \"kubernetes.io/projected/fc97df13-49df-4ee8-bcde-b775f4f594cd-kube-api-access-l5gwg\") pod \"maas-api-revoke-sub-63ecfd7f-vjzz8\" (UID: \"fc97df13-49df-4ee8-bcde-b775f4f594cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63ecfd7f-vjzz8"
Oct 30 21:02:08.097604 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:08.097525    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63ecfd7f-vjzz8"
Oct 30 21:02:08.354492 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:08.354186    4521 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-5tqcb"
Oct 30 21:02:08.355102 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:08.355049    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-5tqcb"
Oct 30 21:02:08.356389 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:08.356347    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-5tqcb"
Oct 30 21:02:08.423250 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:08.422407    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63ecfd7f-vjzz8"]
Oct 30 21:02:08.425875 ip-10-0-131-125 kubenswrapper[4521]: W1030 21:02:08.425825    4521 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc97df13_49df_4ee8_bcde_b775f4f594cd.slice/crio-70df082ba03dd4ce577e1841063f09a627d1efbe711af254e077e73e8bfe74aa WatchSource:0}: Error finding container 70df082ba03dd4ce577e1841063f09a627d1efbe711af254e077e73e8bfe74aa: Status 404 returned error can't find the container with id 70df082ba03dd4ce577e1841063f09a627d1efbe711af254e077e73e8bfe74aa
Oct 30 21:02:08.516404 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:08.516367    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63ecfd7f-vjzz8" event={"ID":"fc97df13-49df-4ee8-bcde-b775f4f594cd","Type":"ContainerStarted","Data":"70df082ba03dd4ce577e1841063f09a627d1efbe711af254e077e73e8bfe74aa"}
Oct 30 21:02:08.597661 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:08.597619    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-39fb96-7954dbb4bc-r8z6k" podUID="f3530279-92bb-445f-95b2-441c22fadc9f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:02:08.800876 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:08.800837    4521 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18ca0a94-c322-41fa-9fd9-0f14419b5075" path="/var/lib/kubelet/pods/18ca0a94-c322-41fa-9fd9-0f14419b5075/volumes"
Oct 30 21:02:09.190622 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:09.190584    4521 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b875f90-b66l4"]
Oct 30 21:02:09.195106 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:09.195084    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b875f90-b66l4"
Oct 30 21:02:09.203112 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:09.203047    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b875f90-b66l4"]
Oct 30 21:02:09.341273 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:09.341234    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c6d7c2b-3302-4066-b0e5-842f5d549b4a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9b875f90-b66l4\" (UID: \"0c6d7c2b-3302-4066-b0e5-842f5d549b4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b875f90-b66l4"
Oct 30 21:02:09.341497 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:09.341467    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77724\" (UniqueName: \"kubernetes.io/projected/0c6d7c2b-3302-4066-b0e5-842f5d549b4a-kube-api-access-77724\") pod \"maas-api-revoke-sub-9b875f90-b66l4\" (UID: \"0c6d7c2b-3302-4066-b0e5-842f5d549b4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b875f90-b66l4"
Oct 30 21:02:09.341731 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:09.341707    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c6d7c2b-3302-4066-b0e5-842f5d549b4a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9b875f90-b66l4\" (UID: \"0c6d7c2b-3302-4066-b0e5-842f5d549b4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b875f90-b66l4"
Oct 30 21:02:09.442700 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:09.442625    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c6d7c2b-3302-4066-b0e5-842f5d549b4a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9b875f90-b66l4\" (UID: \"0c6d7c2b-3302-4066-b0e5-842f5d549b4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b875f90-b66l4"
Oct 30 21:02:09.442867 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:09.442714    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77724\" (UniqueName: \"kubernetes.io/projected/0c6d7c2b-3302-4066-b0e5-842f5d549b4a-kube-api-access-77724\") pod \"maas-api-revoke-sub-9b875f90-b66l4\" (UID: \"0c6d7c2b-3302-4066-b0e5-842f5d549b4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b875f90-b66l4"
Oct 30 21:02:09.442968 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:09.442881    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c6d7c2b-3302-4066-b0e5-842f5d549b4a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9b875f90-b66l4\" (UID: \"0c6d7c2b-3302-4066-b0e5-842f5d549b4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b875f90-b66l4"
Oct 30 21:02:09.444103 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:09.444075    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c6d7c2b-3302-4066-b0e5-842f5d549b4a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9b875f90-b66l4\" (UID: \"0c6d7c2b-3302-4066-b0e5-842f5d549b4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b875f90-b66l4"
Oct 30 21:02:09.445946 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:09.445919    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c6d7c2b-3302-4066-b0e5-842f5d549b4a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9b875f90-b66l4\" (UID: \"0c6d7c2b-3302-4066-b0e5-842f5d549b4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b875f90-b66l4"
Oct 30 21:02:09.455657 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:09.455632    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-77724\" (UniqueName: \"kubernetes.io/projected/0c6d7c2b-3302-4066-b0e5-842f5d549b4a-kube-api-access-77724\") pod \"maas-api-revoke-sub-9b875f90-b66l4\" (UID: \"0c6d7c2b-3302-4066-b0e5-842f5d549b4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b875f90-b66l4"
Oct 30 21:02:09.530966 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:09.530922    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b875f90-b66l4"
Oct 30 21:02:09.533242 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:09.533208    4521 generic.go:356] "Generic (PLEG): container finished" podID="fc97df13-49df-4ee8-bcde-b775f4f594cd" containerID="25ed489b32819722fcf37e60849dffcb8c1c0160dc8b3ead209e18907d7904b5" exitCode=0
Oct 30 21:02:09.534565 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:09.534532    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63ecfd7f-vjzz8" event={"ID":"fc97df13-49df-4ee8-bcde-b775f4f594cd","Type":"ContainerDied","Data":"25ed489b32819722fcf37e60849dffcb8c1c0160dc8b3ead209e18907d7904b5"}
Oct 30 21:02:09.842081 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:09.842031    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b875f90-b66l4"]
Oct 30 21:02:10.212272 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:10.212244    4521 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1948999-jnnvm"]
Oct 30 21:02:10.218787 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:10.218759    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1948999-jnnvm"
Oct 30 21:02:10.229641 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:10.229611    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1948999-jnnvm"]
Oct 30 21:02:10.331028 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:10.330991    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbglr\" (UniqueName: \"kubernetes.io/projected/c1c83c6d-f821-4a73-8747-b8b45e5f346f-kube-api-access-tbglr\") pod \"maas-api-revoke-sub-a1948999-jnnvm\" (UID: \"c1c83c6d-f821-4a73-8747-b8b45e5f346f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1948999-jnnvm"
Oct 30 21:02:10.331246 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:10.331055    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1c83c6d-f821-4a73-8747-b8b45e5f346f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a1948999-jnnvm\" (UID: \"c1c83c6d-f821-4a73-8747-b8b45e5f346f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1948999-jnnvm"
Oct 30 21:02:10.331246 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:10.331144    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1c83c6d-f821-4a73-8747-b8b45e5f346f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a1948999-jnnvm\" (UID: \"c1c83c6d-f821-4a73-8747-b8b45e5f346f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1948999-jnnvm"
Oct 30 21:02:10.432838 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:10.432748    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1c83c6d-f821-4a73-8747-b8b45e5f346f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a1948999-jnnvm\" (UID: \"c1c83c6d-f821-4a73-8747-b8b45e5f346f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1948999-jnnvm"
Oct 30 21:02:10.433018 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:10.432838    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1c83c6d-f821-4a73-8747-b8b45e5f346f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a1948999-jnnvm\" (UID: \"c1c83c6d-f821-4a73-8747-b8b45e5f346f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1948999-jnnvm"
Oct 30 21:02:10.433124 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:10.433104    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbglr\" (UniqueName: \"kubernetes.io/projected/c1c83c6d-f821-4a73-8747-b8b45e5f346f-kube-api-access-tbglr\") pod \"maas-api-revoke-sub-a1948999-jnnvm\" (UID: \"c1c83c6d-f821-4a73-8747-b8b45e5f346f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1948999-jnnvm"
Oct 30 21:02:10.433711 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:10.433678    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1c83c6d-f821-4a73-8747-b8b45e5f346f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a1948999-jnnvm\" (UID: \"c1c83c6d-f821-4a73-8747-b8b45e5f346f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1948999-jnnvm"
Oct 30 21:02:10.435300 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:10.435273    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1c83c6d-f821-4a73-8747-b8b45e5f346f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a1948999-jnnvm\" (UID: \"c1c83c6d-f821-4a73-8747-b8b45e5f346f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1948999-jnnvm"
Oct 30 21:02:10.445724 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:10.445698    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbglr\" (UniqueName: \"kubernetes.io/projected/c1c83c6d-f821-4a73-8747-b8b45e5f346f-kube-api-access-tbglr\") pod \"maas-api-revoke-sub-a1948999-jnnvm\" (UID: \"c1c83c6d-f821-4a73-8747-b8b45e5f346f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1948999-jnnvm"
Oct 30 21:02:10.545402 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:10.545369    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1948999-jnnvm"
Oct 30 21:02:10.547012 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:10.546974    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b875f90-b66l4" event={"ID":"0c6d7c2b-3302-4066-b0e5-842f5d549b4a","Type":"ContainerStarted","Data":"89b11ef9292b842b495a43808bec52f06d5dcca68b1dcb3612d604c6d4314d37"}
Oct 30 21:02:10.547152 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:10.547042    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b875f90-b66l4" event={"ID":"0c6d7c2b-3302-4066-b0e5-842f5d549b4a","Type":"ContainerStarted","Data":"2f128d03af84636d5ca7a6ed495e395b65162c022daafceaca8dc1acd9c467d7"}
Oct 30 21:02:10.747309 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:10.747282    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63ecfd7f-vjzz8"
Oct 30 21:02:10.777257 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:10.777204    4521 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b875f90-b66l4" podStartSLOduration=1.777188344 podStartE2EDuration="1.777188344s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:02:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:02:10.583874389 +0000 UTC m=+1397.929495628" watchObservedRunningTime="2026-10-30 21:02:10.777188344 +0000 UTC m=+1398.122809581"
Oct 30 21:02:10.838710 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:10.838675    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc97df13-49df-4ee8-bcde-b775f4f594cd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc97df13-49df-4ee8-bcde-b775f4f594cd-maas-api-openshift-ca\") pod \"fc97df13-49df-4ee8-bcde-b775f4f594cd\" (UID: \"fc97df13-49df-4ee8-bcde-b775f4f594cd\") "
Oct 30 21:02:10.838885 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:10.838726    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc97df13-49df-4ee8-bcde-b775f4f594cd-kube-api-access-l5gwg\" (UniqueName: \"kubernetes.io/projected/fc97df13-49df-4ee8-bcde-b775f4f594cd-kube-api-access-l5gwg\") pod \"fc97df13-49df-4ee8-bcde-b775f4f594cd\" (UID: \"fc97df13-49df-4ee8-bcde-b775f4f594cd\") "
Oct 30 21:02:10.838885 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:10.838778    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc97df13-49df-4ee8-bcde-b775f4f594cd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc97df13-49df-4ee8-bcde-b775f4f594cd-maas-api-xks-ca\") pod \"fc97df13-49df-4ee8-bcde-b775f4f594cd\" (UID: \"fc97df13-49df-4ee8-bcde-b775f4f594cd\") "
Oct 30 21:02:10.840091 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:10.840065    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc97df13-49df-4ee8-bcde-b775f4f594cd-maas-api-openshift-ca" pod "fc97df13-49df-4ee8-bcde-b775f4f594cd" (UID: "fc97df13-49df-4ee8-bcde-b775f4f594cd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:02:10.841560 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:10.841535    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1948999-jnnvm"]
Oct 30 21:02:10.842424 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:10.842400    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc97df13-49df-4ee8-bcde-b775f4f594cd-maas-api-xks-ca" pod "fc97df13-49df-4ee8-bcde-b775f4f594cd" (UID: "fc97df13-49df-4ee8-bcde-b775f4f594cd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:02:10.842606 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:10.842583    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc97df13-49df-4ee8-bcde-b775f4f594cd-kube-api-access-l5gwg" pod "fc97df13-49df-4ee8-bcde-b775f4f594cd" (UID: "fc97df13-49df-4ee8-bcde-b775f4f594cd"). InnerVolumeSpecName "kube-api-access-l5gwg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:02:10.865310 ip-10-0-131-125 kubenswrapper[4521]: W1030 21:02:10.865274    4521 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1c83c6d_f821_4a73_8747_b8b45e5f346f.slice/crio-a0ece0b7aa6a8d13f15fc60eb4729a2e2150964b9bbc4da3dd40f03f22cccfc0 WatchSource:0}: Error finding container a0ece0b7aa6a8d13f15fc60eb4729a2e2150964b9bbc4da3dd40f03f22cccfc0: Status 404 returned error can't find the container with id a0ece0b7aa6a8d13f15fc60eb4729a2e2150964b9bbc4da3dd40f03f22cccfc0
Oct 30 21:02:10.941615 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:10.941577    4521 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc97df13-49df-4ee8-bcde-b775f4f594cd-maas-api-openshift-ca\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:10.941860 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:10.941833    4521 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5gwg\" (UniqueName: \"kubernetes.io/projected/fc97df13-49df-4ee8-bcde-b775f4f594cd-kube-api-access-l5gwg\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:10.941971 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:10.941863    4521 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc97df13-49df-4ee8-bcde-b775f4f594cd-maas-api-xks-ca\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:11.223841 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:11.223801    4521 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-7mf77"]
Oct 30 21:02:11.224750 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:11.224726    4521 cpu_manager.go:414] "containerMap: removing container" podUID="fc97df13-49df-4ee8-bcde-b775f4f594cd" containerName="revoke-keys"
Oct 30 21:02:11.224831 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:11.224753    4521 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc97df13-49df-4ee8-bcde-b775f4f594cd" containerName="revoke-keys"
Oct 30 21:02:11.224939 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:11.224924    4521 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc97df13-49df-4ee8-bcde-b775f4f594cd" containerName="revoke-keys"
Oct 30 21:02:11.236662 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:11.236631    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-7mf77"
Oct 30 21:02:11.242788 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:11.242760    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-7mf77"]
Oct 30 21:02:11.347363 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:11.347263    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-px4hw\" (UniqueName: \"kubernetes.io/projected/2f940874-3c9d-4de3-985d-5d3c69b0edd3-kube-api-access-px4hw\") pod \"e2e-worker-w1-c4b379-openshift-default-57fdd6d787-7mf77\" (UID: \"2f940874-3c9d-4de3-985d-5d3c69b0edd3\") " pod="openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-7mf77"
Oct 30 21:02:11.347363 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:11.347314    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f940874-3c9d-4de3-985d-5d3c69b0edd3-workload-certs\") pod \"e2e-worker-w1-c4b379-openshift-default-57fdd6d787-7mf77\" (UID: \"2f940874-3c9d-4de3-985d-5d3c69b0edd3\") " pod="openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-7mf77"
Oct 30 21:02:11.347558 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:11.347425    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f940874-3c9d-4de3-985d-5d3c69b0edd3-istiod-ca-cert\") pod \"e2e-worker-w1-c4b379-openshift-default-57fdd6d787-7mf77\" (UID: \"2f940874-3c9d-4de3-985d-5d3c69b0edd3\") " pod="openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-7mf77"
Oct 30 21:02:11.347558 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:11.347476    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f940874-3c9d-4de3-985d-5d3c69b0edd3-istio-data\") pod \"e2e-worker-w1-c4b379-openshift-default-57fdd6d787-7mf77\" (UID: \"2f940874-3c9d-4de3-985d-5d3c69b0edd3\") " pod="openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-7mf77"
Oct 30 21:02:11.347644 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:11.347613    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f940874-3c9d-4de3-985d-5d3c69b0edd3-istio-envoy\") pod \"e2e-worker-w1-c4b379-openshift-default-57fdd6d787-7mf77\" (UID: \"2f940874-3c9d-4de3-985d-5d3c69b0edd3\") " pod="openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-7mf77"
Oct 30 21:02:11.347714 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:11.347698    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f940874-3c9d-4de3-985d-5d3c69b0edd3-workload-socket\") pod \"e2e-worker-w1-c4b379-openshift-default-57fdd6d787-7mf77\" (UID: \"2f940874-3c9d-4de3-985d-5d3c69b0edd3\") " pod="openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-7mf77"
Oct 30 21:02:11.347843 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:11.347824    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f940874-3c9d-4de3-985d-5d3c69b0edd3-credential-socket\") pod \"e2e-worker-w1-c4b379-openshift-default-57fdd6d787-7mf77\" (UID: \"2f940874-3c9d-4de3-985d-5d3c69b0edd3\") " pod="openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-7mf77"
Oct 30 21:02:11.347879 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:11.347857    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f940874-3c9d-4de3-985d-5d3c69b0edd3-istio-token\") pod \"e2e-worker-w1-c4b379-openshift-default-57fdd6d787-7mf77\" (UID: \"2f940874-3c9d-4de3-985d-5d3c69b0edd3\") " pod="openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-7mf77"
Oct 30 21:02:11.347930 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:11.347917    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f940874-3c9d-4de3-985d-5d3c69b0edd3-istio-podinfo\") pod \"e2e-worker-w1-c4b379-openshift-default-57fdd6d787-7mf77\" (UID: \"2f940874-3c9d-4de3-985d-5d3c69b0edd3\") " pod="openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-7mf77"
Oct 30 21:02:11.449340 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:11.449283    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f940874-3c9d-4de3-985d-5d3c69b0edd3-credential-socket\") pod \"e2e-worker-w1-c4b379-openshift-default-57fdd6d787-7mf77\" (UID: \"2f940874-3c9d-4de3-985d-5d3c69b0edd3\") " pod="openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-7mf77"
Oct 30 21:02:11.449495 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:11.449364    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f940874-3c9d-4de3-985d-5d3c69b0edd3-istio-token\") pod \"e2e-worker-w1-c4b379-openshift-default-57fdd6d787-7mf77\" (UID: \"2f940874-3c9d-4de3-985d-5d3c69b0edd3\") " pod="openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-7mf77"
Oct 30 21:02:11.449495 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:11.449413    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f940874-3c9d-4de3-985d-5d3c69b0edd3-istio-podinfo\") pod \"e2e-worker-w1-c4b379-openshift-default-57fdd6d787-7mf77\" (UID: \"2f940874-3c9d-4de3-985d-5d3c69b0edd3\") " pod="openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-7mf77"
Oct 30 21:02:11.449606 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:11.449528    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-px4hw\" (UniqueName: \"kubernetes.io/projected/2f940874-3c9d-4de3-985d-5d3c69b0edd3-kube-api-access-px4hw\") pod \"e2e-worker-w1-c4b379-openshift-default-57fdd6d787-7mf77\" (UID: \"2f940874-3c9d-4de3-985d-5d3c69b0edd3\") " pod="openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-7mf77"
Oct 30 21:02:11.449606 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:11.449571    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f940874-3c9d-4de3-985d-5d3c69b0edd3-workload-certs\") pod \"e2e-worker-w1-c4b379-openshift-default-57fdd6d787-7mf77\" (UID: \"2f940874-3c9d-4de3-985d-5d3c69b0edd3\") " pod="openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-7mf77"
Oct 30 21:02:11.449681 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:11.449615    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f940874-3c9d-4de3-985d-5d3c69b0edd3-istiod-ca-cert\") pod \"e2e-worker-w1-c4b379-openshift-default-57fdd6d787-7mf77\" (UID: \"2f940874-3c9d-4de3-985d-5d3c69b0edd3\") " pod="openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-7mf77"
Oct 30 21:02:11.449720 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:11.449688    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f940874-3c9d-4de3-985d-5d3c69b0edd3-istio-data\") pod \"e2e-worker-w1-c4b379-openshift-default-57fdd6d787-7mf77\" (UID: \"2f940874-3c9d-4de3-985d-5d3c69b0edd3\") " pod="openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-7mf77"
Oct 30 21:02:11.449775 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:11.449757    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f940874-3c9d-4de3-985d-5d3c69b0edd3-istio-envoy\") pod \"e2e-worker-w1-c4b379-openshift-default-57fdd6d787-7mf77\" (UID: \"2f940874-3c9d-4de3-985d-5d3c69b0edd3\") " pod="openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-7mf77"
Oct 30 21:02:11.449809 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:11.449792    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f940874-3c9d-4de3-985d-5d3c69b0edd3-credential-socket\") pod \"e2e-worker-w1-c4b379-openshift-default-57fdd6d787-7mf77\" (UID: \"2f940874-3c9d-4de3-985d-5d3c69b0edd3\") " pod="openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-7mf77"
Oct 30 21:02:11.449845 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:11.449812    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f940874-3c9d-4de3-985d-5d3c69b0edd3-workload-socket\") pod \"e2e-worker-w1-c4b379-openshift-default-57fdd6d787-7mf77\" (UID: \"2f940874-3c9d-4de3-985d-5d3c69b0edd3\") " pod="openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-7mf77"
Oct 30 21:02:11.450114 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:11.450094    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f940874-3c9d-4de3-985d-5d3c69b0edd3-workload-socket\") pod \"e2e-worker-w1-c4b379-openshift-default-57fdd6d787-7mf77\" (UID: \"2f940874-3c9d-4de3-985d-5d3c69b0edd3\") " pod="openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-7mf77"
Oct 30 21:02:11.450151 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:11.450125    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f940874-3c9d-4de3-985d-5d3c69b0edd3-workload-certs\") pod \"e2e-worker-w1-c4b379-openshift-default-57fdd6d787-7mf77\" (UID: \"2f940874-3c9d-4de3-985d-5d3c69b0edd3\") " pod="openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-7mf77"
Oct 30 21:02:11.450406 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:11.450385    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f940874-3c9d-4de3-985d-5d3c69b0edd3-istio-data\") pod \"e2e-worker-w1-c4b379-openshift-default-57fdd6d787-7mf77\" (UID: \"2f940874-3c9d-4de3-985d-5d3c69b0edd3\") " pod="openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-7mf77"
Oct 30 21:02:11.450970 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:11.450949    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f940874-3c9d-4de3-985d-5d3c69b0edd3-istiod-ca-cert\") pod \"e2e-worker-w1-c4b379-openshift-default-57fdd6d787-7mf77\" (UID: \"2f940874-3c9d-4de3-985d-5d3c69b0edd3\") " pod="openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-7mf77"
Oct 30 21:02:11.453082 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:11.453054    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f940874-3c9d-4de3-985d-5d3c69b0edd3-istio-envoy\") pod \"e2e-worker-w1-c4b379-openshift-default-57fdd6d787-7mf77\" (UID: \"2f940874-3c9d-4de3-985d-5d3c69b0edd3\") " pod="openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-7mf77"
Oct 30 21:02:11.454861 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:11.454814    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f940874-3c9d-4de3-985d-5d3c69b0edd3-istio-podinfo\") pod \"e2e-worker-w1-c4b379-openshift-default-57fdd6d787-7mf77\" (UID: \"2f940874-3c9d-4de3-985d-5d3c69b0edd3\") " pod="openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-7mf77"
Oct 30 21:02:11.463994 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:11.463965    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f940874-3c9d-4de3-985d-5d3c69b0edd3-istio-token\") pod \"e2e-worker-w1-c4b379-openshift-default-57fdd6d787-7mf77\" (UID: \"2f940874-3c9d-4de3-985d-5d3c69b0edd3\") " pod="openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-7mf77"
Oct 30 21:02:11.464316 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:11.464287    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-px4hw\" (UniqueName: \"kubernetes.io/projected/2f940874-3c9d-4de3-985d-5d3c69b0edd3-kube-api-access-px4hw\") pod \"e2e-worker-w1-c4b379-openshift-default-57fdd6d787-7mf77\" (UID: \"2f940874-3c9d-4de3-985d-5d3c69b0edd3\") " pod="openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-7mf77"
Oct 30 21:02:11.553584 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:11.553549    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-7mf77"
Oct 30 21:02:11.557987 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:11.557916    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63ecfd7f-vjzz8" event={"ID":"fc97df13-49df-4ee8-bcde-b775f4f594cd","Type":"ContainerDied","Data":"70df082ba03dd4ce577e1841063f09a627d1efbe711af254e077e73e8bfe74aa"}
Oct 30 21:02:11.557987 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:11.557940    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63ecfd7f-vjzz8"
Oct 30 21:02:11.557987 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:11.557967    4521 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="70df082ba03dd4ce577e1841063f09a627d1efbe711af254e077e73e8bfe74aa"
Oct 30 21:02:11.560274 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:11.560244    4521 generic.go:356] "Generic (PLEG): container finished" podID="0c6d7c2b-3302-4066-b0e5-842f5d549b4a" containerID="89b11ef9292b842b495a43808bec52f06d5dcca68b1dcb3612d604c6d4314d37" exitCode=0
Oct 30 21:02:11.560469 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:11.560319    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b875f90-b66l4" event={"ID":"0c6d7c2b-3302-4066-b0e5-842f5d549b4a","Type":"ContainerDied","Data":"89b11ef9292b842b495a43808bec52f06d5dcca68b1dcb3612d604c6d4314d37"}
Oct 30 21:02:11.562607 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:11.562581    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1948999-jnnvm" event={"ID":"c1c83c6d-f821-4a73-8747-b8b45e5f346f","Type":"ContainerStarted","Data":"d4834ace6cb1b60c0405fe5ae3fe13c0d4ef269177a8799eb4f2bd94535177c6"}
Oct 30 21:02:11.562793 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:11.562773    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1948999-jnnvm" event={"ID":"c1c83c6d-f821-4a73-8747-b8b45e5f346f","Type":"ContainerStarted","Data":"a0ece0b7aa6a8d13f15fc60eb4729a2e2150964b9bbc4da3dd40f03f22cccfc0"}
Oct 30 21:02:11.602040 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:11.601938    4521 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1948999-jnnvm" podStartSLOduration=1.601920249 podStartE2EDuration="1.601920249s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:02:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:02:11.599148484 +0000 UTC m=+1398.944769723" watchObservedRunningTime="2026-10-30 21:02:11.601920249 +0000 UTC m=+1398.947541489"
Oct 30 21:02:11.852640 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:11.852583    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-7mf77"]
Oct 30 21:02:11.875937 ip-10-0-131-125 kubenswrapper[4521]: W1030 21:02:11.875890    4521 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f940874_3c9d_4de3_985d_5d3c69b0edd3.slice/crio-359e894924043d1c36d2b7abea7fa1b6405cf671e64827950e4aa7fae91ba3a6 WatchSource:0}: Error finding container 359e894924043d1c36d2b7abea7fa1b6405cf671e64827950e4aa7fae91ba3a6: Status 404 returned error can't find the container with id 359e894924043d1c36d2b7abea7fa1b6405cf671e64827950e4aa7fae91ba3a6
Oct 30 21:02:11.878754 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:11.878712    4521 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:02:11.878862 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:11.878809    4521 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:02:11.878862 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:11.878854    4521 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:02:12.573264 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:12.573224    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-7mf77" event={"ID":"2f940874-3c9d-4de3-985d-5d3c69b0edd3","Type":"ContainerStarted","Data":"4b4499114c447412b06dd2081f5c6ce55e0795bb55f022f9d305b6f629db1afb"}
Oct 30 21:02:12.573728 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:12.573297    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-7mf77" event={"ID":"2f940874-3c9d-4de3-985d-5d3c69b0edd3","Type":"ContainerStarted","Data":"359e894924043d1c36d2b7abea7fa1b6405cf671e64827950e4aa7fae91ba3a6"}
Oct 30 21:02:12.575003 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:12.574978    4521 generic.go:356] "Generic (PLEG): container finished" podID="c1c83c6d-f821-4a73-8747-b8b45e5f346f" containerID="d4834ace6cb1b60c0405fe5ae3fe13c0d4ef269177a8799eb4f2bd94535177c6" exitCode=0
Oct 30 21:02:12.575134 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:12.575062    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1948999-jnnvm" event={"ID":"c1c83c6d-f821-4a73-8747-b8b45e5f346f","Type":"ContainerDied","Data":"d4834ace6cb1b60c0405fe5ae3fe13c0d4ef269177a8799eb4f2bd94535177c6"}
Oct 30 21:02:12.600815 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:12.600762    4521 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-7mf77" podStartSLOduration=1.60074804 podStartE2EDuration="1.60074804s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:02:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:02:12.596309583 +0000 UTC m=+1399.941930847" watchObservedRunningTime="2026-10-30 21:02:12.60074804 +0000 UTC m=+1399.946369342"
Oct 30 21:02:12.819846 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:12.819494    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b875f90-b66l4"
Oct 30 21:02:12.880861 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:12.880769    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c6d7c2b-3302-4066-b0e5-842f5d549b4a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c6d7c2b-3302-4066-b0e5-842f5d549b4a-maas-api-xks-ca\") pod \"0c6d7c2b-3302-4066-b0e5-842f5d549b4a\" (UID: \"0c6d7c2b-3302-4066-b0e5-842f5d549b4a\") "
Oct 30 21:02:12.881244 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:12.881211    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c6d7c2b-3302-4066-b0e5-842f5d549b4a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c6d7c2b-3302-4066-b0e5-842f5d549b4a-maas-api-openshift-ca\") pod \"0c6d7c2b-3302-4066-b0e5-842f5d549b4a\" (UID: \"0c6d7c2b-3302-4066-b0e5-842f5d549b4a\") "
Oct 30 21:02:12.881541 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:12.881518    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c6d7c2b-3302-4066-b0e5-842f5d549b4a-kube-api-access-77724\" (UniqueName: \"kubernetes.io/projected/0c6d7c2b-3302-4066-b0e5-842f5d549b4a-kube-api-access-77724\") pod \"0c6d7c2b-3302-4066-b0e5-842f5d549b4a\" (UID: \"0c6d7c2b-3302-4066-b0e5-842f5d549b4a\") "
Oct 30 21:02:12.884525 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:12.883932    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c6d7c2b-3302-4066-b0e5-842f5d549b4a-maas-api-openshift-ca" pod "0c6d7c2b-3302-4066-b0e5-842f5d549b4a" (UID: "0c6d7c2b-3302-4066-b0e5-842f5d549b4a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:02:12.885640 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:12.885619    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c6d7c2b-3302-4066-b0e5-842f5d549b4a-maas-api-xks-ca" pod "0c6d7c2b-3302-4066-b0e5-842f5d549b4a" (UID: "0c6d7c2b-3302-4066-b0e5-842f5d549b4a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:02:12.889992 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:12.889374    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c6d7c2b-3302-4066-b0e5-842f5d549b4a-kube-api-access-77724" pod "0c6d7c2b-3302-4066-b0e5-842f5d549b4a" (UID: "0c6d7c2b-3302-4066-b0e5-842f5d549b4a"). InnerVolumeSpecName "kube-api-access-77724". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:02:12.984281 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:12.984247    4521 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77724\" (UniqueName: \"kubernetes.io/projected/0c6d7c2b-3302-4066-b0e5-842f5d549b4a-kube-api-access-77724\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:12.984281 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:12.984281    4521 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c6d7c2b-3302-4066-b0e5-842f5d549b4a-maas-api-xks-ca\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:12.984500 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:12.984292    4521 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c6d7c2b-3302-4066-b0e5-842f5d549b4a-maas-api-openshift-ca\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:13.554738 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:13.554705    4521 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-7mf77"
Oct 30 21:02:13.556124 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:13.556089    4521 patch_prober.go:29] interesting pod/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-7mf77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Oct 30 21:02:13.556228 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:13.556161    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-7mf77" podUID="2f940874-3c9d-4de3-985d-5d3c69b0edd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Oct 30 21:02:13.586025 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:13.585996    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b875f90-b66l4"
Oct 30 21:02:13.586482 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:13.586022    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b875f90-b66l4" event={"ID":"0c6d7c2b-3302-4066-b0e5-842f5d549b4a","Type":"ContainerDied","Data":"2f128d03af84636d5ca7a6ed495e395b65162c022daafceaca8dc1acd9c467d7"}
Oct 30 21:02:13.586482 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:13.586072    4521 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2f128d03af84636d5ca7a6ed495e395b65162c022daafceaca8dc1acd9c467d7"
Oct 30 21:02:13.960076 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:13.960045    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1948999-jnnvm"
Oct 30 21:02:13.997932 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:13.997897    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1c83c6d-f821-4a73-8747-b8b45e5f346f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1c83c6d-f821-4a73-8747-b8b45e5f346f-maas-api-openshift-ca\") pod \"c1c83c6d-f821-4a73-8747-b8b45e5f346f\" (UID: \"c1c83c6d-f821-4a73-8747-b8b45e5f346f\") "
Oct 30 21:02:13.998107 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:13.998021    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1c83c6d-f821-4a73-8747-b8b45e5f346f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1c83c6d-f821-4a73-8747-b8b45e5f346f-maas-api-xks-ca\") pod \"c1c83c6d-f821-4a73-8747-b8b45e5f346f\" (UID: \"c1c83c6d-f821-4a73-8747-b8b45e5f346f\") "
Oct 30 21:02:13.998199 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:13.998179    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1c83c6d-f821-4a73-8747-b8b45e5f346f-kube-api-access-tbglr\" (UniqueName: \"kubernetes.io/projected/c1c83c6d-f821-4a73-8747-b8b45e5f346f-kube-api-access-tbglr\") pod \"c1c83c6d-f821-4a73-8747-b8b45e5f346f\" (UID: \"c1c83c6d-f821-4a73-8747-b8b45e5f346f\") "
Oct 30 21:02:13.998376 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:13.998276    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1c83c6d-f821-4a73-8747-b8b45e5f346f-maas-api-openshift-ca" pod "c1c83c6d-f821-4a73-8747-b8b45e5f346f" (UID: "c1c83c6d-f821-4a73-8747-b8b45e5f346f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:02:13.998864 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:13.998840    4521 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1c83c6d-f821-4a73-8747-b8b45e5f346f-maas-api-openshift-ca\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:14.001500 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:14.001472    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1c83c6d-f821-4a73-8747-b8b45e5f346f-maas-api-xks-ca" pod "c1c83c6d-f821-4a73-8747-b8b45e5f346f" (UID: "c1c83c6d-f821-4a73-8747-b8b45e5f346f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:02:14.005830 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:14.005802    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1c83c6d-f821-4a73-8747-b8b45e5f346f-kube-api-access-tbglr" pod "c1c83c6d-f821-4a73-8747-b8b45e5f346f" (UID: "c1c83c6d-f821-4a73-8747-b8b45e5f346f"). InnerVolumeSpecName "kube-api-access-tbglr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:02:14.099841 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:14.099807    4521 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbglr\" (UniqueName: \"kubernetes.io/projected/c1c83c6d-f821-4a73-8747-b8b45e5f346f-kube-api-access-tbglr\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:14.099841 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:14.099839    4521 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1c83c6d-f821-4a73-8747-b8b45e5f346f-maas-api-xks-ca\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:14.554515 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:14.554477    4521 patch_prober.go:29] interesting pod/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-7mf77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Oct 30 21:02:14.554662 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:14.554536    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-7mf77" podUID="2f940874-3c9d-4de3-985d-5d3c69b0edd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Oct 30 21:02:14.596546 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:14.596515    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1948999-jnnvm"
Oct 30 21:02:14.596935 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:14.596542    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1948999-jnnvm" event={"ID":"c1c83c6d-f821-4a73-8747-b8b45e5f346f","Type":"ContainerDied","Data":"a0ece0b7aa6a8d13f15fc60eb4729a2e2150964b9bbc4da3dd40f03f22cccfc0"}
Oct 30 21:02:14.596935 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:14.596590    4521 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a0ece0b7aa6a8d13f15fc60eb4729a2e2150964b9bbc4da3dd40f03f22cccfc0"
Oct 30 21:02:14.632348 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:14.623606    4521 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-4a0c5b/premium-w0-4a0c5b-kserve-85dcf89f57-mscdn"]
Oct 30 21:02:15.554012 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:15.553966    4521 patch_prober.go:29] interesting pod/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-7mf77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Oct 30 21:02:15.554174 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:15.554045    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-7mf77" podUID="2f940874-3c9d-4de3-985d-5d3c69b0edd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Oct 30 21:02:15.933404 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:15.931428    4521 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90fc8705-ckxls"]
Oct 30 21:02:15.933404 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:15.932351    4521 cpu_manager.go:414] "containerMap: removing container" podUID="c1c83c6d-f821-4a73-8747-b8b45e5f346f" containerName="revoke-keys"
Oct 30 21:02:15.933404 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:15.932370    4521 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1c83c6d-f821-4a73-8747-b8b45e5f346f" containerName="revoke-keys"
Oct 30 21:02:15.949195 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:15.947451    4521 cpu_manager.go:414] "containerMap: removing container" podUID="0c6d7c2b-3302-4066-b0e5-842f5d549b4a" containerName="revoke-keys"
Oct 30 21:02:15.949195 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:15.947486    4521 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c6d7c2b-3302-4066-b0e5-842f5d549b4a" containerName="revoke-keys"
Oct 30 21:02:15.949195 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:15.947740    4521 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1c83c6d-f821-4a73-8747-b8b45e5f346f" containerName="revoke-keys"
Oct 30 21:02:15.949195 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:15.947762    4521 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c6d7c2b-3302-4066-b0e5-842f5d549b4a" containerName="revoke-keys"
Oct 30 21:02:15.970346 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:15.969481    4521 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba2d654d-mkl4j"]
Oct 30 21:02:15.970346 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:15.969624    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90fc8705-ckxls"
Oct 30 21:02:15.982538 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:15.982479    4521 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:02:15.984793 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:15.983968    4521 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c45pw\""
Oct 30 21:02:15.997635 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:15.997604    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90fc8705-ckxls"]
Oct 30 21:02:15.997771 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:15.997663    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba2d654d-mkl4j"]
Oct 30 21:02:15.997771 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:15.997766    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba2d654d-mkl4j"
Oct 30 21:02:16.058844 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:16.058814    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2894673f-f88e-4d29-840c-982cd4e217bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ba2d654d-mkl4j\" (UID: \"2894673f-f88e-4d29-840c-982cd4e217bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba2d654d-mkl4j"
Oct 30 21:02:16.059090 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:16.059073    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ls766\" (UniqueName: \"kubernetes.io/projected/2894673f-f88e-4d29-840c-982cd4e217bc-kube-api-access-ls766\") pod \"maas-api-revoke-sub-ba2d654d-mkl4j\" (UID: \"2894673f-f88e-4d29-840c-982cd4e217bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba2d654d-mkl4j"
Oct 30 21:02:16.059254 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:16.059242    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/867c8cde-f176-451a-b65b-939652b2f312-maas-api-xks-ca\") pod \"maas-api-revoke-sub-90fc8705-ckxls\" (UID: \"867c8cde-f176-451a-b65b-939652b2f312\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90fc8705-ckxls"
Oct 30 21:02:16.059398 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:16.059385    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2894673f-f88e-4d29-840c-982cd4e217bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ba2d654d-mkl4j\" (UID: \"2894673f-f88e-4d29-840c-982cd4e217bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba2d654d-mkl4j"
Oct 30 21:02:16.059530 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:16.059514    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rdf5\" (UniqueName: \"kubernetes.io/projected/867c8cde-f176-451a-b65b-939652b2f312-kube-api-access-5rdf5\") pod \"maas-api-revoke-sub-90fc8705-ckxls\" (UID: \"867c8cde-f176-451a-b65b-939652b2f312\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90fc8705-ckxls"
Oct 30 21:02:16.059712 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:16.059693    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/867c8cde-f176-451a-b65b-939652b2f312-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-90fc8705-ckxls\" (UID: \"867c8cde-f176-451a-b65b-939652b2f312\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90fc8705-ckxls"
Oct 30 21:02:16.161170 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:16.161128    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/867c8cde-f176-451a-b65b-939652b2f312-maas-api-xks-ca\") pod \"maas-api-revoke-sub-90fc8705-ckxls\" (UID: \"867c8cde-f176-451a-b65b-939652b2f312\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90fc8705-ckxls"
Oct 30 21:02:16.161170 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:16.161171    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2894673f-f88e-4d29-840c-982cd4e217bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ba2d654d-mkl4j\" (UID: \"2894673f-f88e-4d29-840c-982cd4e217bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba2d654d-mkl4j"
Oct 30 21:02:16.161473 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:16.161223    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rdf5\" (UniqueName: \"kubernetes.io/projected/867c8cde-f176-451a-b65b-939652b2f312-kube-api-access-5rdf5\") pod \"maas-api-revoke-sub-90fc8705-ckxls\" (UID: \"867c8cde-f176-451a-b65b-939652b2f312\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90fc8705-ckxls"
Oct 30 21:02:16.161473 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:16.161378    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/867c8cde-f176-451a-b65b-939652b2f312-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-90fc8705-ckxls\" (UID: \"867c8cde-f176-451a-b65b-939652b2f312\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90fc8705-ckxls"
Oct 30 21:02:16.161573 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:16.161495    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2894673f-f88e-4d29-840c-982cd4e217bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ba2d654d-mkl4j\" (UID: \"2894673f-f88e-4d29-840c-982cd4e217bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba2d654d-mkl4j"
Oct 30 21:02:16.161640 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:16.161615    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ls766\" (UniqueName: \"kubernetes.io/projected/2894673f-f88e-4d29-840c-982cd4e217bc-kube-api-access-ls766\") pod \"maas-api-revoke-sub-ba2d654d-mkl4j\" (UID: \"2894673f-f88e-4d29-840c-982cd4e217bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba2d654d-mkl4j"
Oct 30 21:02:16.162114 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:16.162087    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/867c8cde-f176-451a-b65b-939652b2f312-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-90fc8705-ckxls\" (UID: \"867c8cde-f176-451a-b65b-939652b2f312\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90fc8705-ckxls"
Oct 30 21:02:16.162229 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:16.162126    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2894673f-f88e-4d29-840c-982cd4e217bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ba2d654d-mkl4j\" (UID: \"2894673f-f88e-4d29-840c-982cd4e217bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba2d654d-mkl4j"
Oct 30 21:02:16.163893 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:16.163867    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2894673f-f88e-4d29-840c-982cd4e217bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ba2d654d-mkl4j\" (UID: \"2894673f-f88e-4d29-840c-982cd4e217bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba2d654d-mkl4j"
Oct 30 21:02:16.163968 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:16.163924    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/867c8cde-f176-451a-b65b-939652b2f312-maas-api-xks-ca\") pod \"maas-api-revoke-sub-90fc8705-ckxls\" (UID: \"867c8cde-f176-451a-b65b-939652b2f312\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90fc8705-ckxls"
Oct 30 21:02:16.175225 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:16.175200    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rdf5\" (UniqueName: \"kubernetes.io/projected/867c8cde-f176-451a-b65b-939652b2f312-kube-api-access-5rdf5\") pod \"maas-api-revoke-sub-90fc8705-ckxls\" (UID: \"867c8cde-f176-451a-b65b-939652b2f312\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90fc8705-ckxls"
Oct 30 21:02:16.176371 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:16.176347    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ls766\" (UniqueName: \"kubernetes.io/projected/2894673f-f88e-4d29-840c-982cd4e217bc-kube-api-access-ls766\") pod \"maas-api-revoke-sub-ba2d654d-mkl4j\" (UID: \"2894673f-f88e-4d29-840c-982cd4e217bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba2d654d-mkl4j"
Oct 30 21:02:16.302404 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:16.302366    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90fc8705-ckxls"
Oct 30 21:02:16.355346 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:16.354710    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba2d654d-mkl4j"
Oct 30 21:02:16.558292 ip-10-0-131-125 kubenswrapper[4521]: E1030 21:02:16.556956    4521 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lkn2k" containerName="extractor"
Oct 30 21:02:16.558292 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:16.557046    4521 patch_prober.go:29] interesting pod/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-7mf77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Oct 30 21:02:16.558292 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:16.557102    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-7mf77" podUID="2f940874-3c9d-4de3-985d-5d3c69b0edd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Oct 30 21:02:16.746242 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:16.746215    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90fc8705-ckxls"]
Oct 30 21:02:16.793352 ip-10-0-131-125 kubenswrapper[4521]: W1030 21:02:16.793290    4521 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2894673f_f88e_4d29_840c_982cd4e217bc.slice/crio-be1e09c2824a2d125db287ee7825cb18f53e32896c6365795eaa84150a691b68 WatchSource:0}: Error finding container be1e09c2824a2d125db287ee7825cb18f53e32896c6365795eaa84150a691b68: Status 404 returned error can't find the container with id be1e09c2824a2d125db287ee7825cb18f53e32896c6365795eaa84150a691b68
Oct 30 21:02:16.800772 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:16.796006    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba2d654d-mkl4j"]
Oct 30 21:02:17.554743 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:17.554698    4521 patch_prober.go:29] interesting pod/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-7mf77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Oct 30 21:02:17.555222 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:17.554760    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-7mf77" podUID="2f940874-3c9d-4de3-985d-5d3c69b0edd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Oct 30 21:02:17.644751 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:17.644658    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90fc8705-ckxls" event={"ID":"867c8cde-f176-451a-b65b-939652b2f312","Type":"ContainerStarted","Data":"8a9c2791366a819fceeb64882405b3c156aa337e0b21f32cf95fdc861eefc465"}
Oct 30 21:02:17.644751 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:17.644726    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90fc8705-ckxls" event={"ID":"867c8cde-f176-451a-b65b-939652b2f312","Type":"ContainerStarted","Data":"19fc05fc0f1522405b7cc324f98f21a5653d97da0bc5b0a7a3f9c33a39dee504"}
Oct 30 21:02:17.646668 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:17.646638    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba2d654d-mkl4j" event={"ID":"2894673f-f88e-4d29-840c-982cd4e217bc","Type":"ContainerStarted","Data":"c9b38afcfda0af7dba2226ce02ebc4ca7a37be554534fdfe24d8bb9587f744e3"}
Oct 30 21:02:17.646781 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:17.646693    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba2d654d-mkl4j" event={"ID":"2894673f-f88e-4d29-840c-982cd4e217bc","Type":"ContainerStarted","Data":"be1e09c2824a2d125db287ee7825cb18f53e32896c6365795eaa84150a691b68"}
Oct 30 21:02:17.679459 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:17.679400    4521 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90fc8705-ckxls" podStartSLOduration=2.679383625 podStartE2EDuration="2.679383625s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:02:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:02:17.673737645 +0000 UTC m=+1405.019358900" watchObservedRunningTime="2026-10-30 21:02:17.679383625 +0000 UTC m=+1405.025004863"
Oct 30 21:02:17.746618 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:17.746548    4521 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba2d654d-mkl4j" podStartSLOduration=2.746523867 podStartE2EDuration="2.746523867s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:02:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:02:17.724592916 +0000 UTC m=+1405.070214154" watchObservedRunningTime="2026-10-30 21:02:17.746523867 +0000 UTC m=+1405.092145106"
Oct 30 21:02:18.484430 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:18.484134    4521 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-32a969/simulator-w2-32a969-kserve-6cc8dfc44c-dwhlt" podUID="62a6adf7-d17a-4fec-a707-a2d549b67753" containerName="main" containerID="cri-o://d0cc9665eaaba2041a8be6760b20929fefb206724c54b74e9da703a83acecc51" gracePeriod=42
Oct 30 21:02:18.494647 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:18.494603    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-32a969/simulator-w2-32a969-kserve-6cc8dfc44c-dwhlt" podUID="62a6adf7-d17a-4fec-a707-a2d549b67753" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": EOF"
Oct 30 21:02:18.554636 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:18.554549    4521 patch_prober.go:29] interesting pod/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-7mf77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Oct 30 21:02:18.554636 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:18.554618    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-7mf77" podUID="2f940874-3c9d-4de3-985d-5d3c69b0edd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Oct 30 21:02:18.622717 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:18.622684    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-39fb96-7954dbb4bc-r8z6k"
Oct 30 21:02:18.665382 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:18.663492    4521 generic.go:356] "Generic (PLEG): container finished" podID="62a6adf7-d17a-4fec-a707-a2d549b67753" containerID="d0cc9665eaaba2041a8be6760b20929fefb206724c54b74e9da703a83acecc51" exitCode=0
Oct 30 21:02:18.665382 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:18.663600    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-32a969/simulator-w2-32a969-kserve-6cc8dfc44c-dwhlt" event={"ID":"62a6adf7-d17a-4fec-a707-a2d549b67753","Type":"ContainerDied","Data":"d0cc9665eaaba2041a8be6760b20929fefb206724c54b74e9da703a83acecc51"}
Oct 30 21:02:18.672239 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:18.672208    4521 generic.go:356] "Generic (PLEG): container finished" podID="f3530279-92bb-445f-95b2-441c22fadc9f" containerID="57592b57782433ff180994a4f8417e3267dac98c3d770d97e6aad2722facb942" exitCode=0
Oct 30 21:02:18.672468 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:18.672444    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-39fb96-7954dbb4bc-r8z6k" event={"ID":"f3530279-92bb-445f-95b2-441c22fadc9f","Type":"ContainerDied","Data":"57592b57782433ff180994a4f8417e3267dac98c3d770d97e6aad2722facb942"}
Oct 30 21:02:18.672566 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:18.672509    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-39fb96-7954dbb4bc-r8z6k" event={"ID":"f3530279-92bb-445f-95b2-441c22fadc9f","Type":"ContainerDied","Data":"b8a1b2d998fd5eaa5f4db5ca8125dd56822b6823a4bda8cf0af600434a781d17"}
Oct 30 21:02:18.672566 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:18.672543    4521 scope.go:122] "RemoveContainer" containerID="57592b57782433ff180994a4f8417e3267dac98c3d770d97e6aad2722facb942"
Oct 30 21:02:18.673163 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:18.672705    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-39fb96-7954dbb4bc-r8z6k"
Oct 30 21:02:18.675508 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:18.674817    4521 generic.go:356] "Generic (PLEG): container finished" podID="2894673f-f88e-4d29-840c-982cd4e217bc" containerID="c9b38afcfda0af7dba2226ce02ebc4ca7a37be554534fdfe24d8bb9587f744e3" exitCode=0
Oct 30 21:02:18.675508 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:18.674987    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba2d654d-mkl4j" event={"ID":"2894673f-f88e-4d29-840c-982cd4e217bc","Type":"ContainerDied","Data":"c9b38afcfda0af7dba2226ce02ebc4ca7a37be554534fdfe24d8bb9587f744e3"}
Oct 30 21:02:18.677228 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:18.677205    4521 generic.go:356] "Generic (PLEG): container finished" podID="867c8cde-f176-451a-b65b-939652b2f312" containerID="8a9c2791366a819fceeb64882405b3c156aa337e0b21f32cf95fdc861eefc465" exitCode=0
Oct 30 21:02:18.677315 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:18.677238    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90fc8705-ckxls" event={"ID":"867c8cde-f176-451a-b65b-939652b2f312","Type":"ContainerDied","Data":"8a9c2791366a819fceeb64882405b3c156aa337e0b21f32cf95fdc861eefc465"}
Oct 30 21:02:18.690292 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:18.690192    4521 scope.go:122] "RemoveContainer" containerID="57592b57782433ff180994a4f8417e3267dac98c3d770d97e6aad2722facb942"
Oct 30 21:02:18.690635 ip-10-0-131-125 kubenswrapper[4521]: E1030 21:02:18.690599    4521 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57592b57782433ff180994a4f8417e3267dac98c3d770d97e6aad2722facb942\": container with ID starting with 57592b57782433ff180994a4f8417e3267dac98c3d770d97e6aad2722facb942 not found: ID does not exist" containerID="57592b57782433ff180994a4f8417e3267dac98c3d770d97e6aad2722facb942"
Oct 30 21:02:18.690726 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:18.690640    4521 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57592b57782433ff180994a4f8417e3267dac98c3d770d97e6aad2722facb942"} err="failed to get container status \"57592b57782433ff180994a4f8417e3267dac98c3d770d97e6aad2722facb942\": rpc error: code = NotFound desc = could not find container \"57592b57782433ff180994a4f8417e3267dac98c3d770d97e6aad2722facb942\": container with ID starting with 57592b57782433ff180994a4f8417e3267dac98c3d770d97e6aad2722facb942 not found: ID does not exist"
Oct 30 21:02:18.708082 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:18.708047    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3530279-92bb-445f-95b2-441c22fadc9f-kube-api-access-9dv47\" (UniqueName: \"kubernetes.io/projected/f3530279-92bb-445f-95b2-441c22fadc9f-kube-api-access-9dv47\") pod \"f3530279-92bb-445f-95b2-441c22fadc9f\" (UID: \"f3530279-92bb-445f-95b2-441c22fadc9f\") "
Oct 30 21:02:18.708308 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:18.708289    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3530279-92bb-445f-95b2-441c22fadc9f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3530279-92bb-445f-95b2-441c22fadc9f-plugins-config-volume\") pod \"f3530279-92bb-445f-95b2-441c22fadc9f\" (UID: \"f3530279-92bb-445f-95b2-441c22fadc9f\") "
Oct 30 21:02:18.708644 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:18.708621    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3530279-92bb-445f-95b2-441c22fadc9f-plugins-config-volume" pod "f3530279-92bb-445f-95b2-441c22fadc9f" (UID: "f3530279-92bb-445f-95b2-441c22fadc9f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:02:18.708894 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:18.708874    4521 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3530279-92bb-445f-95b2-441c22fadc9f-plugins-config-volume\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:18.710900 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:18.710876    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3530279-92bb-445f-95b2-441c22fadc9f-kube-api-access-9dv47" pod "f3530279-92bb-445f-95b2-441c22fadc9f" (UID: "f3530279-92bb-445f-95b2-441c22fadc9f"). InnerVolumeSpecName "kube-api-access-9dv47". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:02:18.812428 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:18.812399    4521 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9dv47\" (UniqueName: \"kubernetes.io/projected/f3530279-92bb-445f-95b2-441c22fadc9f-kube-api-access-9dv47\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:18.823850 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:18.823819    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-32a969/simulator-w2-32a969-kserve-6cc8dfc44c-dwhlt"
Oct 30 21:02:18.932347 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:18.931462    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62a6adf7-d17a-4fec-a707-a2d549b67753-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62a6adf7-d17a-4fec-a707-a2d549b67753-tmp-dir\") pod \"62a6adf7-d17a-4fec-a707-a2d549b67753\" (UID: \"62a6adf7-d17a-4fec-a707-a2d549b67753\") "
Oct 30 21:02:18.932347 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:18.931644    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62a6adf7-d17a-4fec-a707-a2d549b67753-kube-api-access-2hk2j\" (UniqueName: \"kubernetes.io/projected/62a6adf7-d17a-4fec-a707-a2d549b67753-kube-api-access-2hk2j\") pod \"62a6adf7-d17a-4fec-a707-a2d549b67753\" (UID: \"62a6adf7-d17a-4fec-a707-a2d549b67753\") "
Oct 30 21:02:18.932347 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:18.931693    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62a6adf7-d17a-4fec-a707-a2d549b67753-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62a6adf7-d17a-4fec-a707-a2d549b67753-model-cache\") pod \"62a6adf7-d17a-4fec-a707-a2d549b67753\" (UID: \"62a6adf7-d17a-4fec-a707-a2d549b67753\") "
Oct 30 21:02:18.932347 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:18.931778    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62a6adf7-d17a-4fec-a707-a2d549b67753-home\" (UniqueName: \"kubernetes.io/empty-dir/62a6adf7-d17a-4fec-a707-a2d549b67753-home\") pod \"62a6adf7-d17a-4fec-a707-a2d549b67753\" (UID: \"62a6adf7-d17a-4fec-a707-a2d549b67753\") "
Oct 30 21:02:18.932347 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:18.931825    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62a6adf7-d17a-4fec-a707-a2d549b67753-tmp-dir" pod "62a6adf7-d17a-4fec-a707-a2d549b67753" (UID: "62a6adf7-d17a-4fec-a707-a2d549b67753"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:02:18.932347 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:18.932059    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62a6adf7-d17a-4fec-a707-a2d549b67753-dshm\" (UniqueName: \"kubernetes.io/empty-dir/62a6adf7-d17a-4fec-a707-a2d549b67753-dshm\") pod \"62a6adf7-d17a-4fec-a707-a2d549b67753\" (UID: \"62a6adf7-d17a-4fec-a707-a2d549b67753\") "
Oct 30 21:02:18.932347 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:18.932105    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/62a6adf7-d17a-4fec-a707-a2d549b67753-tls-certs\" (UniqueName: \"kubernetes.io/secret/62a6adf7-d17a-4fec-a707-a2d549b67753-tls-certs\") pod \"62a6adf7-d17a-4fec-a707-a2d549b67753\" (UID: \"62a6adf7-d17a-4fec-a707-a2d549b67753\") "
Oct 30 21:02:18.932813 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:18.932387    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62a6adf7-d17a-4fec-a707-a2d549b67753-home" pod "62a6adf7-d17a-4fec-a707-a2d549b67753" (UID: "62a6adf7-d17a-4fec-a707-a2d549b67753"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:02:18.932813 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:18.932578    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62a6adf7-d17a-4fec-a707-a2d549b67753-model-cache" pod "62a6adf7-d17a-4fec-a707-a2d549b67753" (UID: "62a6adf7-d17a-4fec-a707-a2d549b67753"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:02:18.932986 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:18.932955    4521 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62a6adf7-d17a-4fec-a707-a2d549b67753-tmp-dir\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:18.933158 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:18.933142    4521 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62a6adf7-d17a-4fec-a707-a2d549b67753-model-cache\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:18.933347 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:18.933246    4521 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62a6adf7-d17a-4fec-a707-a2d549b67753-home\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:18.936537 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:18.936246    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62a6adf7-d17a-4fec-a707-a2d549b67753-tls-certs" pod "62a6adf7-d17a-4fec-a707-a2d549b67753" (UID: "62a6adf7-d17a-4fec-a707-a2d549b67753"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:02:18.938348 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:18.938306    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62a6adf7-d17a-4fec-a707-a2d549b67753-dshm" pod "62a6adf7-d17a-4fec-a707-a2d549b67753" (UID: "62a6adf7-d17a-4fec-a707-a2d549b67753"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:02:18.941978 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:18.941944    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62a6adf7-d17a-4fec-a707-a2d549b67753-kube-api-access-2hk2j" pod "62a6adf7-d17a-4fec-a707-a2d549b67753" (UID: "62a6adf7-d17a-4fec-a707-a2d549b67753"). InnerVolumeSpecName "kube-api-access-2hk2j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:02:19.002135 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:19.002101    4521 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-39fb96-7954dbb4bc-r8z6k"]
Oct 30 21:02:19.018727 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:19.018695    4521 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-39fb96-7954dbb4bc-r8z6k"]
Oct 30 21:02:19.035186 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:19.035147    4521 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62a6adf7-d17a-4fec-a707-a2d549b67753-dshm\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:19.035186 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:19.035190    4521 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62a6adf7-d17a-4fec-a707-a2d549b67753-tls-certs\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:19.035401 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:19.035206    4521 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hk2j\" (UniqueName: \"kubernetes.io/projected/62a6adf7-d17a-4fec-a707-a2d549b67753-kube-api-access-2hk2j\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:19.559676 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:19.559603    4521 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-7mf77"
Oct 30 21:02:19.560113 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:19.560095    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-7mf77"
Oct 30 21:02:19.560954 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:19.560939    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-7mf77"
Oct 30 21:02:19.595030 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:19.594979    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-39fb96-7954dbb4bc-r8z6k" podUID="f3530279-92bb-445f-95b2-441c22fadc9f" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.47:50052\" within 1s: context deadline exceeded"
Oct 30 21:02:19.595183 ip-10-0-131-125 kubenswrapper[4521]: W1030 21:02:19.595048    4521 logging.go:55] [core] [Channel #267 SubChannel #270]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.47:50052", ServerName: "10.133.0.47:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.47:50052: operation was canceled"
Oct 30 21:02:19.687376 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:19.687311    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-32a969/simulator-w2-32a969-kserve-6cc8dfc44c-dwhlt" event={"ID":"62a6adf7-d17a-4fec-a707-a2d549b67753","Type":"ContainerDied","Data":"ef10ad04f689458a8bcd631fda475ad89f00e222b8ec93f2ffbf197f2aafebf0"}
Oct 30 21:02:19.687376 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:19.687366    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-32a969/simulator-w2-32a969-kserve-6cc8dfc44c-dwhlt"
Oct 30 21:02:19.687754 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:19.687402    4521 scope.go:122] "RemoveContainer" containerID="d0cc9665eaaba2041a8be6760b20929fefb206724c54b74e9da703a83acecc51"
Oct 30 21:02:19.721277 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:19.721199    4521 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-32a969/simulator-w2-32a969-kserve-6cc8dfc44c-dwhlt"]
Oct 30 21:02:19.726077 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:19.724537    4521 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-32a969/simulator-w2-32a969-kserve-6cc8dfc44c-dwhlt"]
Oct 30 21:02:19.901446 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:19.901046    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba2d654d-mkl4j"
Oct 30 21:02:19.930213 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:19.930185    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90fc8705-ckxls"
Oct 30 21:02:19.951024 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:19.950989    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2894673f-f88e-4d29-840c-982cd4e217bc-kube-api-access-ls766\" (UniqueName: \"kubernetes.io/projected/2894673f-f88e-4d29-840c-982cd4e217bc-kube-api-access-ls766\") pod \"2894673f-f88e-4d29-840c-982cd4e217bc\" (UID: \"2894673f-f88e-4d29-840c-982cd4e217bc\") "
Oct 30 21:02:19.951203 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:19.951132    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2894673f-f88e-4d29-840c-982cd4e217bc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2894673f-f88e-4d29-840c-982cd4e217bc-maas-api-xks-ca\") pod \"2894673f-f88e-4d29-840c-982cd4e217bc\" (UID: \"2894673f-f88e-4d29-840c-982cd4e217bc\") "
Oct 30 21:02:19.951268 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:19.951241    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2894673f-f88e-4d29-840c-982cd4e217bc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2894673f-f88e-4d29-840c-982cd4e217bc-maas-api-openshift-ca\") pod \"2894673f-f88e-4d29-840c-982cd4e217bc\" (UID: \"2894673f-f88e-4d29-840c-982cd4e217bc\") "
Oct 30 21:02:19.952147 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:19.952117    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2894673f-f88e-4d29-840c-982cd4e217bc-maas-api-openshift-ca" pod "2894673f-f88e-4d29-840c-982cd4e217bc" (UID: "2894673f-f88e-4d29-840c-982cd4e217bc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:02:19.954068 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:19.954046    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2894673f-f88e-4d29-840c-982cd4e217bc-kube-api-access-ls766" pod "2894673f-f88e-4d29-840c-982cd4e217bc" (UID: "2894673f-f88e-4d29-840c-982cd4e217bc"). InnerVolumeSpecName "kube-api-access-ls766". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:02:19.954840 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:19.954803    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2894673f-f88e-4d29-840c-982cd4e217bc-maas-api-xks-ca" pod "2894673f-f88e-4d29-840c-982cd4e217bc" (UID: "2894673f-f88e-4d29-840c-982cd4e217bc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:02:19.999917 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:19.999862    4521 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-65968b-hfxjz"]
Oct 30 21:02:20.000728 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:20.000702    4521 cpu_manager.go:414] "containerMap: removing container" podUID="62a6adf7-d17a-4fec-a707-a2d549b67753" containerName="main"
Oct 30 21:02:20.000728 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:20.000726    4521 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62a6adf7-d17a-4fec-a707-a2d549b67753" containerName="main"
Oct 30 21:02:20.000898 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:20.000742    4521 cpu_manager.go:414] "containerMap: removing container" podUID="867c8cde-f176-451a-b65b-939652b2f312" containerName="revoke-keys"
Oct 30 21:02:20.000898 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:20.000752    4521 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="867c8cde-f176-451a-b65b-939652b2f312" containerName="revoke-keys"
Oct 30 21:02:20.000898 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:20.000768    4521 cpu_manager.go:414] "containerMap: removing container" podUID="f3530279-92bb-445f-95b2-441c22fadc9f" containerName="payload-pre-processing"
Oct 30 21:02:20.000898 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:20.000775    4521 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3530279-92bb-445f-95b2-441c22fadc9f" containerName="payload-pre-processing"
Oct 30 21:02:20.000898 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:20.000791    4521 cpu_manager.go:414] "containerMap: removing container" podUID="2894673f-f88e-4d29-840c-982cd4e217bc" containerName="revoke-keys"
Oct 30 21:02:20.000898 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:20.000798    4521 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2894673f-f88e-4d29-840c-982cd4e217bc" containerName="revoke-keys"
Oct 30 21:02:20.001173 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:20.000988    4521 memory_manager.go:365] "RemoveStaleState removing state" podUID="62a6adf7-d17a-4fec-a707-a2d549b67753" containerName="main"
Oct 30 21:02:20.001173 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:20.001002    4521 memory_manager.go:365] "RemoveStaleState removing state" podUID="867c8cde-f176-451a-b65b-939652b2f312" containerName="revoke-keys"
Oct 30 21:02:20.001173 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:20.001014    4521 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3530279-92bb-445f-95b2-441c22fadc9f" containerName="payload-pre-processing"
Oct 30 21:02:20.001173 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:20.001025    4521 memory_manager.go:365] "RemoveStaleState removing state" podUID="2894673f-f88e-4d29-840c-982cd4e217bc" containerName="revoke-keys"
Oct 30 21:02:20.004926 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:20.004906    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-65968b-hfxjz"
Oct 30 21:02:20.020380 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:20.020291    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-65968b-hfxjz"]
Oct 30 21:02:20.052648 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:20.052611    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/867c8cde-f176-451a-b65b-939652b2f312-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/867c8cde-f176-451a-b65b-939652b2f312-maas-api-xks-ca\") pod \"867c8cde-f176-451a-b65b-939652b2f312\" (UID: \"867c8cde-f176-451a-b65b-939652b2f312\") "
Oct 30 21:02:20.052835 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:20.052676    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/867c8cde-f176-451a-b65b-939652b2f312-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/867c8cde-f176-451a-b65b-939652b2f312-maas-api-openshift-ca\") pod \"867c8cde-f176-451a-b65b-939652b2f312\" (UID: \"867c8cde-f176-451a-b65b-939652b2f312\") "
Oct 30 21:02:20.052935 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:20.052917    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/867c8cde-f176-451a-b65b-939652b2f312-kube-api-access-5rdf5\" (UniqueName: \"kubernetes.io/projected/867c8cde-f176-451a-b65b-939652b2f312-kube-api-access-5rdf5\") pod \"867c8cde-f176-451a-b65b-939652b2f312\" (UID: \"867c8cde-f176-451a-b65b-939652b2f312\") "
Oct 30 21:02:20.053135 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:20.053107    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/867c8cde-f176-451a-b65b-939652b2f312-maas-api-openshift-ca" pod "867c8cde-f176-451a-b65b-939652b2f312" (UID: "867c8cde-f176-451a-b65b-939652b2f312"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:02:20.053225 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:20.053143    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1078ebe9-650f-4a2c-9347-dee3fd88e4de-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-65968b-hfxjz\" (UID: \"1078ebe9-650f-4a2c-9347-dee3fd88e4de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-65968b-hfxjz"
Oct 30 21:02:20.053280 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:20.053237    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1078ebe9-650f-4a2c-9347-dee3fd88e4de-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-65968b-hfxjz\" (UID: \"1078ebe9-650f-4a2c-9347-dee3fd88e4de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-65968b-hfxjz"
Oct 30 21:02:20.053497 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:20.053468    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhrp8\" (UniqueName: \"kubernetes.io/projected/1078ebe9-650f-4a2c-9347-dee3fd88e4de-kube-api-access-qhrp8\") pod \"maas-api-revoke-keys-e2e-worker-w5-65968b-hfxjz\" (UID: \"1078ebe9-650f-4a2c-9347-dee3fd88e4de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-65968b-hfxjz"
Oct 30 21:02:20.053744 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:20.053719    4521 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2894673f-f88e-4d29-840c-982cd4e217bc-maas-api-xks-ca\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:20.053798 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:20.053757    4521 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2894673f-f88e-4d29-840c-982cd4e217bc-maas-api-openshift-ca\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:20.053798 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:20.053775    4521 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/867c8cde-f176-451a-b65b-939652b2f312-maas-api-openshift-ca\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:20.053798 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:20.053792    4521 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ls766\" (UniqueName: \"kubernetes.io/projected/2894673f-f88e-4d29-840c-982cd4e217bc-kube-api-access-ls766\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:20.056175 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:20.056144    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/867c8cde-f176-451a-b65b-939652b2f312-maas-api-xks-ca" pod "867c8cde-f176-451a-b65b-939652b2f312" (UID: "867c8cde-f176-451a-b65b-939652b2f312"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:02:20.056804 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:20.056778    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/867c8cde-f176-451a-b65b-939652b2f312-kube-api-access-5rdf5" pod "867c8cde-f176-451a-b65b-939652b2f312" (UID: "867c8cde-f176-451a-b65b-939652b2f312"). InnerVolumeSpecName "kube-api-access-5rdf5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:02:20.156972 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:20.156880    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1078ebe9-650f-4a2c-9347-dee3fd88e4de-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-65968b-hfxjz\" (UID: \"1078ebe9-650f-4a2c-9347-dee3fd88e4de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-65968b-hfxjz"
Oct 30 21:02:20.157125 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:20.157043    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhrp8\" (UniqueName: \"kubernetes.io/projected/1078ebe9-650f-4a2c-9347-dee3fd88e4de-kube-api-access-qhrp8\") pod \"maas-api-revoke-keys-e2e-worker-w5-65968b-hfxjz\" (UID: \"1078ebe9-650f-4a2c-9347-dee3fd88e4de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-65968b-hfxjz"
Oct 30 21:02:20.157687 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:20.157231    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1078ebe9-650f-4a2c-9347-dee3fd88e4de-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-65968b-hfxjz\" (UID: \"1078ebe9-650f-4a2c-9347-dee3fd88e4de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-65968b-hfxjz"
Oct 30 21:02:20.157687 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:20.157293    4521 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rdf5\" (UniqueName: \"kubernetes.io/projected/867c8cde-f176-451a-b65b-939652b2f312-kube-api-access-5rdf5\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:20.157687 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:20.157309    4521 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/867c8cde-f176-451a-b65b-939652b2f312-maas-api-xks-ca\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:20.158466 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:20.158437    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1078ebe9-650f-4a2c-9347-dee3fd88e4de-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-65968b-hfxjz\" (UID: \"1078ebe9-650f-4a2c-9347-dee3fd88e4de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-65968b-hfxjz"
Oct 30 21:02:20.161182 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:20.161136    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1078ebe9-650f-4a2c-9347-dee3fd88e4de-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-65968b-hfxjz\" (UID: \"1078ebe9-650f-4a2c-9347-dee3fd88e4de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-65968b-hfxjz"
Oct 30 21:02:20.172316 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:20.172270    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhrp8\" (UniqueName: \"kubernetes.io/projected/1078ebe9-650f-4a2c-9347-dee3fd88e4de-kube-api-access-qhrp8\") pod \"maas-api-revoke-keys-e2e-worker-w5-65968b-hfxjz\" (UID: \"1078ebe9-650f-4a2c-9347-dee3fd88e4de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-65968b-hfxjz"
Oct 30 21:02:20.328406 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:20.328371    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-65968b-hfxjz"
Oct 30 21:02:20.668612 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:20.659279    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-65968b-hfxjz"]
Oct 30 21:02:20.675378 ip-10-0-131-125 kubenswrapper[4521]: W1030 21:02:20.674808    4521 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1078ebe9_650f_4a2c_9347_dee3fd88e4de.slice/crio-83ba764b83e0e81a878d351b9ab3645dc10b847d12c8cfc13fec46a96b84ef89 WatchSource:0}: Error finding container 83ba764b83e0e81a878d351b9ab3645dc10b847d12c8cfc13fec46a96b84ef89: Status 404 returned error can't find the container with id 83ba764b83e0e81a878d351b9ab3645dc10b847d12c8cfc13fec46a96b84ef89
Oct 30 21:02:20.701289 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:20.701270    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba2d654d-mkl4j"
Oct 30 21:02:20.701477 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:20.701302    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba2d654d-mkl4j" event={"ID":"2894673f-f88e-4d29-840c-982cd4e217bc","Type":"ContainerDied","Data":"be1e09c2824a2d125db287ee7825cb18f53e32896c6365795eaa84150a691b68"}
Oct 30 21:02:20.701544 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:20.701502    4521 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="be1e09c2824a2d125db287ee7825cb18f53e32896c6365795eaa84150a691b68"
Oct 30 21:02:20.702941 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:20.702921    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90fc8705-ckxls"
Oct 30 21:02:20.703061 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:20.702911    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90fc8705-ckxls" event={"ID":"867c8cde-f176-451a-b65b-939652b2f312","Type":"ContainerDied","Data":"19fc05fc0f1522405b7cc324f98f21a5653d97da0bc5b0a7a3f9c33a39dee504"}
Oct 30 21:02:20.703116 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:20.703059    4521 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="19fc05fc0f1522405b7cc324f98f21a5653d97da0bc5b0a7a3f9c33a39dee504"
Oct 30 21:02:20.704460 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:20.704436    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-65968b-hfxjz" event={"ID":"1078ebe9-650f-4a2c-9347-dee3fd88e4de","Type":"ContainerStarted","Data":"83ba764b83e0e81a878d351b9ab3645dc10b847d12c8cfc13fec46a96b84ef89"}
Oct 30 21:02:20.793758 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:20.793723    4521 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62a6adf7-d17a-4fec-a707-a2d549b67753" path="/var/lib/kubelet/pods/62a6adf7-d17a-4fec-a707-a2d549b67753/volumes"
Oct 30 21:02:20.794209 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:20.794194    4521 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3530279-92bb-445f-95b2-441c22fadc9f" path="/var/lib/kubelet/pods/f3530279-92bb-445f-95b2-441c22fadc9f/volumes"
Oct 30 21:02:20.883126 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:20.883072    4521 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-32a969/distinct-2-w2-32a969-kserve-5c7845448b-vt5gw" podUID="d430ed01-65d1-43a6-a0c4-ab82212376a9" containerName="main" containerID="cri-o://66aa776bf0145a82d05e5842d09817cbf3e9d4e1b43e9253569b980f73af2e3a" gracePeriod=40
Oct 30 21:02:20.959113 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:20.959068    4521 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-32a969/distinct-w2-32a969-kserve-7f798b664f-jdh82" podUID="887f1d27-bf24-4b57-b6e4-8fb095df7a50" containerName="main" containerID="cri-o://46781044a82ac81762140ed527675663b0e3cda210817c2aa5dbb28004f922b6" gracePeriod=40
Oct 30 21:02:21.076676 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:21.076640    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-32a969/distinct-w2-32a969-kserve-7f798b664f-jdh82" podUID="887f1d27-bf24-4b57-b6e4-8fb095df7a50" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": dial tcp 10.133.0.72:8000: connect: connection refused"
Oct 30 21:02:21.244082 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:21.244055    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-32a969/distinct-2-w2-32a969-kserve-5c7845448b-vt5gw"
Oct 30 21:02:21.258465 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:21.258437    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-32a969/distinct-w2-32a969-kserve-7f798b664f-jdh82"
Oct 30 21:02:21.396590 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:21.396554    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d430ed01-65d1-43a6-a0c4-ab82212376a9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d430ed01-65d1-43a6-a0c4-ab82212376a9-tmp-dir\") pod \"d430ed01-65d1-43a6-a0c4-ab82212376a9\" (UID: \"d430ed01-65d1-43a6-a0c4-ab82212376a9\") "
Oct 30 21:02:21.396810 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:21.396700    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/887f1d27-bf24-4b57-b6e4-8fb095df7a50-dshm\" (UniqueName: \"kubernetes.io/empty-dir/887f1d27-bf24-4b57-b6e4-8fb095df7a50-dshm\") pod \"887f1d27-bf24-4b57-b6e4-8fb095df7a50\" (UID: \"887f1d27-bf24-4b57-b6e4-8fb095df7a50\") "
Oct 30 21:02:21.396884 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:21.396845    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d430ed01-65d1-43a6-a0c4-ab82212376a9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d430ed01-65d1-43a6-a0c4-ab82212376a9-dshm\") pod \"d430ed01-65d1-43a6-a0c4-ab82212376a9\" (UID: \"d430ed01-65d1-43a6-a0c4-ab82212376a9\") "
Oct 30 21:02:21.396937 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:21.396904    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/887f1d27-bf24-4b57-b6e4-8fb095df7a50-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/887f1d27-bf24-4b57-b6e4-8fb095df7a50-model-cache\") pod \"887f1d27-bf24-4b57-b6e4-8fb095df7a50\" (UID: \"887f1d27-bf24-4b57-b6e4-8fb095df7a50\") "
Oct 30 21:02:21.396998 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:21.396955    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d430ed01-65d1-43a6-a0c4-ab82212376a9-kube-api-access-9sbbt\" (UniqueName: \"kubernetes.io/projected/d430ed01-65d1-43a6-a0c4-ab82212376a9-kube-api-access-9sbbt\") pod \"d430ed01-65d1-43a6-a0c4-ab82212376a9\" (UID: \"d430ed01-65d1-43a6-a0c4-ab82212376a9\") "
Oct 30 21:02:21.397063 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:21.397007    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/887f1d27-bf24-4b57-b6e4-8fb095df7a50-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/887f1d27-bf24-4b57-b6e4-8fb095df7a50-tmp-dir\") pod \"887f1d27-bf24-4b57-b6e4-8fb095df7a50\" (UID: \"887f1d27-bf24-4b57-b6e4-8fb095df7a50\") "
Oct 30 21:02:21.397063 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:21.397033    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d430ed01-65d1-43a6-a0c4-ab82212376a9-home\" (UniqueName: \"kubernetes.io/empty-dir/d430ed01-65d1-43a6-a0c4-ab82212376a9-home\") pod \"d430ed01-65d1-43a6-a0c4-ab82212376a9\" (UID: \"d430ed01-65d1-43a6-a0c4-ab82212376a9\") "
Oct 30 21:02:21.397155 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:21.397131    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/887f1d27-bf24-4b57-b6e4-8fb095df7a50-tls-certs\" (UniqueName: \"kubernetes.io/secret/887f1d27-bf24-4b57-b6e4-8fb095df7a50-tls-certs\") pod \"887f1d27-bf24-4b57-b6e4-8fb095df7a50\" (UID: \"887f1d27-bf24-4b57-b6e4-8fb095df7a50\") "
Oct 30 21:02:21.397207 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:21.397148    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d430ed01-65d1-43a6-a0c4-ab82212376a9-tmp-dir" pod "d430ed01-65d1-43a6-a0c4-ab82212376a9" (UID: "d430ed01-65d1-43a6-a0c4-ab82212376a9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:02:21.397261 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:21.397250    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d430ed01-65d1-43a6-a0c4-ab82212376a9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d430ed01-65d1-43a6-a0c4-ab82212376a9-model-cache\") pod \"d430ed01-65d1-43a6-a0c4-ab82212376a9\" (UID: \"d430ed01-65d1-43a6-a0c4-ab82212376a9\") "
Oct 30 21:02:21.398523 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:21.397384    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/887f1d27-bf24-4b57-b6e4-8fb095df7a50-home\" (UniqueName: \"kubernetes.io/empty-dir/887f1d27-bf24-4b57-b6e4-8fb095df7a50-home\") pod \"887f1d27-bf24-4b57-b6e4-8fb095df7a50\" (UID: \"887f1d27-bf24-4b57-b6e4-8fb095df7a50\") "
Oct 30 21:02:21.398523 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:21.397418    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/887f1d27-bf24-4b57-b6e4-8fb095df7a50-kube-api-access-mvs47\" (UniqueName: \"kubernetes.io/projected/887f1d27-bf24-4b57-b6e4-8fb095df7a50-kube-api-access-mvs47\") pod \"887f1d27-bf24-4b57-b6e4-8fb095df7a50\" (UID: \"887f1d27-bf24-4b57-b6e4-8fb095df7a50\") "
Oct 30 21:02:21.398523 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:21.397458    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d430ed01-65d1-43a6-a0c4-ab82212376a9-tls-certs\" (UniqueName: \"kubernetes.io/secret/d430ed01-65d1-43a6-a0c4-ab82212376a9-tls-certs\") pod \"d430ed01-65d1-43a6-a0c4-ab82212376a9\" (UID: \"d430ed01-65d1-43a6-a0c4-ab82212376a9\") "
Oct 30 21:02:21.398523 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:21.397499    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/887f1d27-bf24-4b57-b6e4-8fb095df7a50-tmp-dir" pod "887f1d27-bf24-4b57-b6e4-8fb095df7a50" (UID: "887f1d27-bf24-4b57-b6e4-8fb095df7a50"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:02:21.398523 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:21.398076    4521 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/887f1d27-bf24-4b57-b6e4-8fb095df7a50-tmp-dir\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:21.398523 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:21.398097    4521 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d430ed01-65d1-43a6-a0c4-ab82212376a9-tmp-dir\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:21.398523 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:21.398094    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/887f1d27-bf24-4b57-b6e4-8fb095df7a50-model-cache" pod "887f1d27-bf24-4b57-b6e4-8fb095df7a50" (UID: "887f1d27-bf24-4b57-b6e4-8fb095df7a50"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:02:21.398523 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:21.398170    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/887f1d27-bf24-4b57-b6e4-8fb095df7a50-home" pod "887f1d27-bf24-4b57-b6e4-8fb095df7a50" (UID: "887f1d27-bf24-4b57-b6e4-8fb095df7a50"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:02:21.398523 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:21.398347    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d430ed01-65d1-43a6-a0c4-ab82212376a9-model-cache" pod "d430ed01-65d1-43a6-a0c4-ab82212376a9" (UID: "d430ed01-65d1-43a6-a0c4-ab82212376a9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:02:21.398523 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:21.398490    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d430ed01-65d1-43a6-a0c4-ab82212376a9-home" pod "d430ed01-65d1-43a6-a0c4-ab82212376a9" (UID: "d430ed01-65d1-43a6-a0c4-ab82212376a9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:02:21.399789 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:21.399734    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d430ed01-65d1-43a6-a0c4-ab82212376a9-dshm" pod "d430ed01-65d1-43a6-a0c4-ab82212376a9" (UID: "d430ed01-65d1-43a6-a0c4-ab82212376a9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:02:21.401463 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:21.401303    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d430ed01-65d1-43a6-a0c4-ab82212376a9-kube-api-access-9sbbt" pod "d430ed01-65d1-43a6-a0c4-ab82212376a9" (UID: "d430ed01-65d1-43a6-a0c4-ab82212376a9"). InnerVolumeSpecName "kube-api-access-9sbbt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:02:21.402975 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:21.402950    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/887f1d27-bf24-4b57-b6e4-8fb095df7a50-dshm" pod "887f1d27-bf24-4b57-b6e4-8fb095df7a50" (UID: "887f1d27-bf24-4b57-b6e4-8fb095df7a50"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:02:21.403086 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:21.402992    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/887f1d27-bf24-4b57-b6e4-8fb095df7a50-tls-certs" pod "887f1d27-bf24-4b57-b6e4-8fb095df7a50" (UID: "887f1d27-bf24-4b57-b6e4-8fb095df7a50"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:02:21.415255 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:21.415170    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/887f1d27-bf24-4b57-b6e4-8fb095df7a50-kube-api-access-mvs47" pod "887f1d27-bf24-4b57-b6e4-8fb095df7a50" (UID: "887f1d27-bf24-4b57-b6e4-8fb095df7a50"). InnerVolumeSpecName "kube-api-access-mvs47". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:02:21.416996 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:21.416907    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d430ed01-65d1-43a6-a0c4-ab82212376a9-tls-certs" pod "d430ed01-65d1-43a6-a0c4-ab82212376a9" (UID: "d430ed01-65d1-43a6-a0c4-ab82212376a9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:02:21.499412 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:21.499301    4521 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d430ed01-65d1-43a6-a0c4-ab82212376a9-model-cache\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:21.499412 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:21.499350    4521 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/887f1d27-bf24-4b57-b6e4-8fb095df7a50-home\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:21.499412 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:21.499362    4521 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvs47\" (UniqueName: \"kubernetes.io/projected/887f1d27-bf24-4b57-b6e4-8fb095df7a50-kube-api-access-mvs47\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:21.499412 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:21.499376    4521 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d430ed01-65d1-43a6-a0c4-ab82212376a9-tls-certs\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:21.499412 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:21.499385    4521 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/887f1d27-bf24-4b57-b6e4-8fb095df7a50-dshm\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:21.499412 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:21.499392    4521 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d430ed01-65d1-43a6-a0c4-ab82212376a9-dshm\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:21.499412 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:21.499400    4521 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/887f1d27-bf24-4b57-b6e4-8fb095df7a50-model-cache\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:21.499412 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:21.499413    4521 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9sbbt\" (UniqueName: \"kubernetes.io/projected/d430ed01-65d1-43a6-a0c4-ab82212376a9-kube-api-access-9sbbt\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:21.499903 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:21.499427    4521 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d430ed01-65d1-43a6-a0c4-ab82212376a9-home\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:21.499903 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:21.499439    4521 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/887f1d27-bf24-4b57-b6e4-8fb095df7a50-tls-certs\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:21.717070 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:21.717038    4521 generic.go:356] "Generic (PLEG): container finished" podID="1078ebe9-650f-4a2c-9347-dee3fd88e4de" containerID="9fd1240d2ea9a4377d88ba3c064b8ee0e27bbf31b95caeafb9ccf062c6800b7f" exitCode=0
Oct 30 21:02:21.717569 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:21.717123    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-65968b-hfxjz" event={"ID":"1078ebe9-650f-4a2c-9347-dee3fd88e4de","Type":"ContainerDied","Data":"9fd1240d2ea9a4377d88ba3c064b8ee0e27bbf31b95caeafb9ccf062c6800b7f"}
Oct 30 21:02:21.718642 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:21.718616    4521 generic.go:356] "Generic (PLEG): container finished" podID="887f1d27-bf24-4b57-b6e4-8fb095df7a50" containerID="46781044a82ac81762140ed527675663b0e3cda210817c2aa5dbb28004f922b6" exitCode=0
Oct 30 21:02:21.718767 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:21.718670    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-32a969/distinct-w2-32a969-kserve-7f798b664f-jdh82"
Oct 30 21:02:21.718767 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:21.718700    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-32a969/distinct-w2-32a969-kserve-7f798b664f-jdh82" event={"ID":"887f1d27-bf24-4b57-b6e4-8fb095df7a50","Type":"ContainerDied","Data":"46781044a82ac81762140ed527675663b0e3cda210817c2aa5dbb28004f922b6"}
Oct 30 21:02:21.718767 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:21.718744    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-32a969/distinct-w2-32a969-kserve-7f798b664f-jdh82" event={"ID":"887f1d27-bf24-4b57-b6e4-8fb095df7a50","Type":"ContainerDied","Data":"d61100423af8a92e7821e7250379d5e8a3b7a7d9802b449ad1729b9a5f6aac6d"}
Oct 30 21:02:21.718767 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:21.718766    4521 scope.go:122] "RemoveContainer" containerID="46781044a82ac81762140ed527675663b0e3cda210817c2aa5dbb28004f922b6"
Oct 30 21:02:21.723510 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:21.721836    4521 generic.go:356] "Generic (PLEG): container finished" podID="d430ed01-65d1-43a6-a0c4-ab82212376a9" containerID="66aa776bf0145a82d05e5842d09817cbf3e9d4e1b43e9253569b980f73af2e3a" exitCode=0
Oct 30 21:02:21.723510 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:21.721894    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-32a969/distinct-2-w2-32a969-kserve-5c7845448b-vt5gw" event={"ID":"d430ed01-65d1-43a6-a0c4-ab82212376a9","Type":"ContainerDied","Data":"66aa776bf0145a82d05e5842d09817cbf3e9d4e1b43e9253569b980f73af2e3a"}
Oct 30 21:02:21.723510 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:21.721923    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-32a969/distinct-2-w2-32a969-kserve-5c7845448b-vt5gw" event={"ID":"d430ed01-65d1-43a6-a0c4-ab82212376a9","Type":"ContainerDied","Data":"57330fffeb199ff8e3bca7ddbe80a07100580731ba35b750a638aaa8022b13f5"}
Oct 30 21:02:21.723510 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:21.721984    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-32a969/distinct-2-w2-32a969-kserve-5c7845448b-vt5gw"
Oct 30 21:02:21.768537 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:21.768512    4521 scope.go:122] "RemoveContainer" containerID="46781044a82ac81762140ed527675663b0e3cda210817c2aa5dbb28004f922b6"
Oct 30 21:02:21.768858 ip-10-0-131-125 kubenswrapper[4521]: E1030 21:02:21.768835    4521 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46781044a82ac81762140ed527675663b0e3cda210817c2aa5dbb28004f922b6\": container with ID starting with 46781044a82ac81762140ed527675663b0e3cda210817c2aa5dbb28004f922b6 not found: ID does not exist" containerID="46781044a82ac81762140ed527675663b0e3cda210817c2aa5dbb28004f922b6"
Oct 30 21:02:21.768925 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:21.768872    4521 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46781044a82ac81762140ed527675663b0e3cda210817c2aa5dbb28004f922b6"} err="failed to get container status \"46781044a82ac81762140ed527675663b0e3cda210817c2aa5dbb28004f922b6\": rpc error: code = NotFound desc = could not find container \"46781044a82ac81762140ed527675663b0e3cda210817c2aa5dbb28004f922b6\": container with ID starting with 46781044a82ac81762140ed527675663b0e3cda210817c2aa5dbb28004f922b6 not found: ID does not exist"
Oct 30 21:02:21.768925 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:21.768897    4521 scope.go:122] "RemoveContainer" containerID="66aa776bf0145a82d05e5842d09817cbf3e9d4e1b43e9253569b980f73af2e3a"
Oct 30 21:02:21.779275 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:21.779253    4521 scope.go:122] "RemoveContainer" containerID="66aa776bf0145a82d05e5842d09817cbf3e9d4e1b43e9253569b980f73af2e3a"
Oct 30 21:02:21.779655 ip-10-0-131-125 kubenswrapper[4521]: E1030 21:02:21.779629    4521 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66aa776bf0145a82d05e5842d09817cbf3e9d4e1b43e9253569b980f73af2e3a\": container with ID starting with 66aa776bf0145a82d05e5842d09817cbf3e9d4e1b43e9253569b980f73af2e3a not found: ID does not exist" containerID="66aa776bf0145a82d05e5842d09817cbf3e9d4e1b43e9253569b980f73af2e3a"
Oct 30 21:02:21.779742 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:21.779666    4521 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66aa776bf0145a82d05e5842d09817cbf3e9d4e1b43e9253569b980f73af2e3a"} err="failed to get container status \"66aa776bf0145a82d05e5842d09817cbf3e9d4e1b43e9253569b980f73af2e3a\": rpc error: code = NotFound desc = could not find container \"66aa776bf0145a82d05e5842d09817cbf3e9d4e1b43e9253569b980f73af2e3a\": container with ID starting with 66aa776bf0145a82d05e5842d09817cbf3e9d4e1b43e9253569b980f73af2e3a not found: ID does not exist"
Oct 30 21:02:21.784505 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:21.784483    4521 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-32a969/distinct-2-w2-32a969-kserve-5c7845448b-vt5gw"]
Oct 30 21:02:21.789349 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:21.789311    4521 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-32a969/distinct-2-w2-32a969-kserve-5c7845448b-vt5gw"]
Oct 30 21:02:21.804055 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:21.804032    4521 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-32a969/distinct-w2-32a969-kserve-7f798b664f-jdh82"]
Oct 30 21:02:21.808441 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:21.808419    4521 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-32a969/distinct-w2-32a969-kserve-7f798b664f-jdh82"]
Oct 30 21:02:21.880459 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:21.880424    4521 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-39fb96-759bc7c496-29tpq"]
Oct 30 21:02:21.880959 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:21.880923    4521 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-39fb96-759bc7c496-29tpq" podUID="9a9ccb4e-15bd-4fe3-8f85-45ab99eea028" containerName="maas-api" containerID="cri-o://411f92977fe45223c42ae4f05363d09a9d2c0760480fe50f22d1f1b7421158e9" gracePeriod=30
Oct 30 21:02:22.406805 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:22.406772    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-39fb96-759bc7c496-29tpq"
Oct 30 21:02:22.537093 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:22.536998    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a9ccb4e-15bd-4fe3-8f85-45ab99eea028-kube-api-access-hz8tl\" (UniqueName: \"kubernetes.io/projected/9a9ccb4e-15bd-4fe3-8f85-45ab99eea028-kube-api-access-hz8tl\") pod \"9a9ccb4e-15bd-4fe3-8f85-45ab99eea028\" (UID: \"9a9ccb4e-15bd-4fe3-8f85-45ab99eea028\") "
Oct 30 21:02:22.537093 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:22.537040    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a9ccb4e-15bd-4fe3-8f85-45ab99eea028-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a9ccb4e-15bd-4fe3-8f85-45ab99eea028-openshift-service-ca\") pod \"9a9ccb4e-15bd-4fe3-8f85-45ab99eea028\" (UID: \"9a9ccb4e-15bd-4fe3-8f85-45ab99eea028\") "
Oct 30 21:02:22.537308 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:22.537167    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9a9ccb4e-15bd-4fe3-8f85-45ab99eea028-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a9ccb4e-15bd-4fe3-8f85-45ab99eea028-metrics-certs\") pod \"9a9ccb4e-15bd-4fe3-8f85-45ab99eea028\" (UID: \"9a9ccb4e-15bd-4fe3-8f85-45ab99eea028\") "
Oct 30 21:02:22.537308 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:22.537210    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9a9ccb4e-15bd-4fe3-8f85-45ab99eea028-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a9ccb4e-15bd-4fe3-8f85-45ab99eea028-maas-api-tls\") pod \"9a9ccb4e-15bd-4fe3-8f85-45ab99eea028\" (UID: \"9a9ccb4e-15bd-4fe3-8f85-45ab99eea028\") "
Oct 30 21:02:22.537597 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:22.537565    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a9ccb4e-15bd-4fe3-8f85-45ab99eea028-openshift-service-ca" pod "9a9ccb4e-15bd-4fe3-8f85-45ab99eea028" (UID: "9a9ccb4e-15bd-4fe3-8f85-45ab99eea028"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:02:22.537970 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:22.537944    4521 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a9ccb4e-15bd-4fe3-8f85-45ab99eea028-openshift-service-ca\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:22.539542 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:22.539512    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a9ccb4e-15bd-4fe3-8f85-45ab99eea028-maas-api-tls" pod "9a9ccb4e-15bd-4fe3-8f85-45ab99eea028" (UID: "9a9ccb4e-15bd-4fe3-8f85-45ab99eea028"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:02:22.539636 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:22.539539    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a9ccb4e-15bd-4fe3-8f85-45ab99eea028-kube-api-access-hz8tl" pod "9a9ccb4e-15bd-4fe3-8f85-45ab99eea028" (UID: "9a9ccb4e-15bd-4fe3-8f85-45ab99eea028"). InnerVolumeSpecName "kube-api-access-hz8tl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:02:22.539710 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:22.539681    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a9ccb4e-15bd-4fe3-8f85-45ab99eea028-metrics-certs" pod "9a9ccb4e-15bd-4fe3-8f85-45ab99eea028" (UID: "9a9ccb4e-15bd-4fe3-8f85-45ab99eea028"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:02:22.639354 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:22.639303    4521 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hz8tl\" (UniqueName: \"kubernetes.io/projected/9a9ccb4e-15bd-4fe3-8f85-45ab99eea028-kube-api-access-hz8tl\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:22.639354 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:22.639351    4521 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a9ccb4e-15bd-4fe3-8f85-45ab99eea028-metrics-certs\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:22.639633 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:22.639365    4521 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a9ccb4e-15bd-4fe3-8f85-45ab99eea028-maas-api-tls\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:22.733829 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:22.733791    4521 generic.go:356] "Generic (PLEG): container finished" podID="9a9ccb4e-15bd-4fe3-8f85-45ab99eea028" containerID="411f92977fe45223c42ae4f05363d09a9d2c0760480fe50f22d1f1b7421158e9" exitCode=0
Oct 30 21:02:22.734279 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:22.733857    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-39fb96-759bc7c496-29tpq"
Oct 30 21:02:22.734279 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:22.733880    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-39fb96-759bc7c496-29tpq" event={"ID":"9a9ccb4e-15bd-4fe3-8f85-45ab99eea028","Type":"ContainerDied","Data":"411f92977fe45223c42ae4f05363d09a9d2c0760480fe50f22d1f1b7421158e9"}
Oct 30 21:02:22.734279 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:22.733948    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-39fb96-759bc7c496-29tpq" event={"ID":"9a9ccb4e-15bd-4fe3-8f85-45ab99eea028","Type":"ContainerDied","Data":"6e1584435ac2130e63215b186ee004cb2b26a6565c919649918ee55b67737534"}
Oct 30 21:02:22.734279 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:22.733981    4521 scope.go:122] "RemoveContainer" containerID="411f92977fe45223c42ae4f05363d09a9d2c0760480fe50f22d1f1b7421158e9"
Oct 30 21:02:22.747767 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:22.747739    4521 scope.go:122] "RemoveContainer" containerID="411f92977fe45223c42ae4f05363d09a9d2c0760480fe50f22d1f1b7421158e9"
Oct 30 21:02:22.748114 ip-10-0-131-125 kubenswrapper[4521]: E1030 21:02:22.748087    4521 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"411f92977fe45223c42ae4f05363d09a9d2c0760480fe50f22d1f1b7421158e9\": container with ID starting with 411f92977fe45223c42ae4f05363d09a9d2c0760480fe50f22d1f1b7421158e9 not found: ID does not exist" containerID="411f92977fe45223c42ae4f05363d09a9d2c0760480fe50f22d1f1b7421158e9"
Oct 30 21:02:22.748190 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:22.748127    4521 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"411f92977fe45223c42ae4f05363d09a9d2c0760480fe50f22d1f1b7421158e9"} err="failed to get container status \"411f92977fe45223c42ae4f05363d09a9d2c0760480fe50f22d1f1b7421158e9\": rpc error: code = NotFound desc = could not find container \"411f92977fe45223c42ae4f05363d09a9d2c0760480fe50f22d1f1b7421158e9\": container with ID starting with 411f92977fe45223c42ae4f05363d09a9d2c0760480fe50f22d1f1b7421158e9 not found: ID does not exist"
Oct 30 21:02:22.769233 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:22.768441    4521 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-39fb96-759bc7c496-29tpq"]
Oct 30 21:02:22.778808 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:22.778705    4521 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-39fb96-759bc7c496-29tpq"]
Oct 30 21:02:22.795669 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:22.795579    4521 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="887f1d27-bf24-4b57-b6e4-8fb095df7a50" path="/var/lib/kubelet/pods/887f1d27-bf24-4b57-b6e4-8fb095df7a50/volumes"
Oct 30 21:02:22.796317 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:22.796293    4521 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a9ccb4e-15bd-4fe3-8f85-45ab99eea028" path="/var/lib/kubelet/pods/9a9ccb4e-15bd-4fe3-8f85-45ab99eea028/volumes"
Oct 30 21:02:22.796960 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:22.796940    4521 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d430ed01-65d1-43a6-a0c4-ab82212376a9" path="/var/lib/kubelet/pods/d430ed01-65d1-43a6-a0c4-ab82212376a9/volumes"
Oct 30 21:02:22.886394 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:22.886365    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-65968b-hfxjz"
Oct 30 21:02:22.943074 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:22.943030    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1078ebe9-650f-4a2c-9347-dee3fd88e4de-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1078ebe9-650f-4a2c-9347-dee3fd88e4de-maas-api-xks-ca\") pod \"1078ebe9-650f-4a2c-9347-dee3fd88e4de\" (UID: \"1078ebe9-650f-4a2c-9347-dee3fd88e4de\") "
Oct 30 21:02:22.943252 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:22.943199    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1078ebe9-650f-4a2c-9347-dee3fd88e4de-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1078ebe9-650f-4a2c-9347-dee3fd88e4de-maas-api-openshift-ca\") pod \"1078ebe9-650f-4a2c-9347-dee3fd88e4de\" (UID: \"1078ebe9-650f-4a2c-9347-dee3fd88e4de\") "
Oct 30 21:02:22.943307 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:22.943286    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1078ebe9-650f-4a2c-9347-dee3fd88e4de-kube-api-access-qhrp8\" (UniqueName: \"kubernetes.io/projected/1078ebe9-650f-4a2c-9347-dee3fd88e4de-kube-api-access-qhrp8\") pod \"1078ebe9-650f-4a2c-9347-dee3fd88e4de\" (UID: \"1078ebe9-650f-4a2c-9347-dee3fd88e4de\") "
Oct 30 21:02:22.943697 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:22.943665    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1078ebe9-650f-4a2c-9347-dee3fd88e4de-maas-api-openshift-ca" pod "1078ebe9-650f-4a2c-9347-dee3fd88e4de" (UID: "1078ebe9-650f-4a2c-9347-dee3fd88e4de"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:02:22.943958 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:22.943938    4521 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1078ebe9-650f-4a2c-9347-dee3fd88e4de-maas-api-openshift-ca\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:22.945576 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:22.945552    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1078ebe9-650f-4a2c-9347-dee3fd88e4de-kube-api-access-qhrp8" pod "1078ebe9-650f-4a2c-9347-dee3fd88e4de" (UID: "1078ebe9-650f-4a2c-9347-dee3fd88e4de"). InnerVolumeSpecName "kube-api-access-qhrp8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:02:22.945979 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:22.945950    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1078ebe9-650f-4a2c-9347-dee3fd88e4de-maas-api-xks-ca" pod "1078ebe9-650f-4a2c-9347-dee3fd88e4de" (UID: "1078ebe9-650f-4a2c-9347-dee3fd88e4de"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:02:23.044926 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:23.044889    4521 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhrp8\" (UniqueName: \"kubernetes.io/projected/1078ebe9-650f-4a2c-9347-dee3fd88e4de-kube-api-access-qhrp8\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:23.044926 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:23.044921    4521 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1078ebe9-650f-4a2c-9347-dee3fd88e4de-maas-api-xks-ca\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:23.748541 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:23.748513    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-65968b-hfxjz"
Oct 30 21:02:23.749036 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:23.748535    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-65968b-hfxjz" event={"ID":"1078ebe9-650f-4a2c-9347-dee3fd88e4de","Type":"ContainerDied","Data":"83ba764b83e0e81a878d351b9ab3645dc10b847d12c8cfc13fec46a96b84ef89"}
Oct 30 21:02:23.749036 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:23.748590    4521 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="83ba764b83e0e81a878d351b9ab3645dc10b847d12c8cfc13fec46a96b84ef89"
Oct 30 21:02:23.782066 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:23.782035    4521 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c2d1eb-9m5xb"]
Oct 30 21:02:23.782666 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:23.782649    4521 cpu_manager.go:414] "containerMap: removing container" podUID="9a9ccb4e-15bd-4fe3-8f85-45ab99eea028" containerName="maas-api"
Oct 30 21:02:23.782767 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:23.782669    4521 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a9ccb4e-15bd-4fe3-8f85-45ab99eea028" containerName="maas-api"
Oct 30 21:02:23.782767 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:23.782689    4521 cpu_manager.go:414] "containerMap: removing container" podUID="887f1d27-bf24-4b57-b6e4-8fb095df7a50" containerName="main"
Oct 30 21:02:23.782767 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:23.782696    4521 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="887f1d27-bf24-4b57-b6e4-8fb095df7a50" containerName="main"
Oct 30 21:02:23.782767 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:23.782746    4521 cpu_manager.go:414] "containerMap: removing container" podUID="1078ebe9-650f-4a2c-9347-dee3fd88e4de" containerName="revoke-keys"
Oct 30 21:02:23.782767 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:23.782754    4521 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1078ebe9-650f-4a2c-9347-dee3fd88e4de" containerName="revoke-keys"
Oct 30 21:02:23.783048 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:23.782776    4521 cpu_manager.go:414] "containerMap: removing container" podUID="d430ed01-65d1-43a6-a0c4-ab82212376a9" containerName="main"
Oct 30 21:02:23.783048 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:23.782785    4521 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d430ed01-65d1-43a6-a0c4-ab82212376a9" containerName="main"
Oct 30 21:02:23.783048 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:23.782927    4521 memory_manager.go:365] "RemoveStaleState removing state" podUID="1078ebe9-650f-4a2c-9347-dee3fd88e4de" containerName="revoke-keys"
Oct 30 21:02:23.783048 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:23.782942    4521 memory_manager.go:365] "RemoveStaleState removing state" podUID="d430ed01-65d1-43a6-a0c4-ab82212376a9" containerName="main"
Oct 30 21:02:23.783048 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:23.782954    4521 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a9ccb4e-15bd-4fe3-8f85-45ab99eea028" containerName="maas-api"
Oct 30 21:02:23.783048 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:23.782966    4521 memory_manager.go:365] "RemoveStaleState removing state" podUID="887f1d27-bf24-4b57-b6e4-8fb095df7a50" containerName="main"
Oct 30 21:02:23.786902 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:23.786879    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c2d1eb-9m5xb"
Oct 30 21:02:23.793482 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:23.793458    4521 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:02:23.794088 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:23.794063    4521 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c45pw\""
Oct 30 21:02:23.795711 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:23.795682    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c2d1eb-9m5xb"]
Oct 30 21:02:23.855980 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:23.855939    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28e05ac8-5b4d-4085-bf8c-58793ca1c370-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-c2d1eb-9m5xb\" (UID: \"28e05ac8-5b4d-4085-bf8c-58793ca1c370\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c2d1eb-9m5xb"
Oct 30 21:02:23.856162 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:23.856121    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ggzh\" (UniqueName: \"kubernetes.io/projected/28e05ac8-5b4d-4085-bf8c-58793ca1c370-kube-api-access-2ggzh\") pod \"maas-api-revoke-keys-e2e-worker-w4-c2d1eb-9m5xb\" (UID: \"28e05ac8-5b4d-4085-bf8c-58793ca1c370\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c2d1eb-9m5xb"
Oct 30 21:02:23.856272 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:23.856254    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28e05ac8-5b4d-4085-bf8c-58793ca1c370-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-c2d1eb-9m5xb\" (UID: \"28e05ac8-5b4d-4085-bf8c-58793ca1c370\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c2d1eb-9m5xb"
Oct 30 21:02:23.957994 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:23.957948    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ggzh\" (UniqueName: \"kubernetes.io/projected/28e05ac8-5b4d-4085-bf8c-58793ca1c370-kube-api-access-2ggzh\") pod \"maas-api-revoke-keys-e2e-worker-w4-c2d1eb-9m5xb\" (UID: \"28e05ac8-5b4d-4085-bf8c-58793ca1c370\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c2d1eb-9m5xb"
Oct 30 21:02:23.958175 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:23.958058    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28e05ac8-5b4d-4085-bf8c-58793ca1c370-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-c2d1eb-9m5xb\" (UID: \"28e05ac8-5b4d-4085-bf8c-58793ca1c370\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c2d1eb-9m5xb"
Oct 30 21:02:23.958175 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:23.958111    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28e05ac8-5b4d-4085-bf8c-58793ca1c370-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-c2d1eb-9m5xb\" (UID: \"28e05ac8-5b4d-4085-bf8c-58793ca1c370\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c2d1eb-9m5xb"
Oct 30 21:02:23.958791 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:23.958765    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28e05ac8-5b4d-4085-bf8c-58793ca1c370-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-c2d1eb-9m5xb\" (UID: \"28e05ac8-5b4d-4085-bf8c-58793ca1c370\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c2d1eb-9m5xb"
Oct 30 21:02:23.960601 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:23.960581    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28e05ac8-5b4d-4085-bf8c-58793ca1c370-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-c2d1eb-9m5xb\" (UID: \"28e05ac8-5b4d-4085-bf8c-58793ca1c370\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c2d1eb-9m5xb"
Oct 30 21:02:23.970415 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:23.970386    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ggzh\" (UniqueName: \"kubernetes.io/projected/28e05ac8-5b4d-4085-bf8c-58793ca1c370-kube-api-access-2ggzh\") pod \"maas-api-revoke-keys-e2e-worker-w4-c2d1eb-9m5xb\" (UID: \"28e05ac8-5b4d-4085-bf8c-58793ca1c370\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c2d1eb-9m5xb"
Oct 30 21:02:24.105391 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:24.105281    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c2d1eb-9m5xb"
Oct 30 21:02:24.167553 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:24.167480    4521 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-214c7679-bnjxj"]
Oct 30 21:02:24.191141 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:24.177027    4521 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-5a8f94/distinct-2-w3-5a8f94-kserve-ffcc6449b-hv48k"]
Oct 30 21:02:24.191141 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:24.177533    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-214c7679-bnjxj"
Oct 30 21:02:24.191141 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:24.190222    4521 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-5a8f94/distinct-w3-5a8f94-kserve-5f8b544d59-mf95j"]
Oct 30 21:02:24.201716 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:24.201652    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-214c7679-bnjxj"]
Oct 30 21:02:24.210555 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:24.210492    4521 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-5a8f94/premium-w3-5a8f94-kserve-bb96b6848-5hk8h"]
Oct 30 21:02:24.224337 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:24.223308    4521 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-5a8f94/simulator-w3-5a8f94-kserve-77cc88cc89-n4sl6"]
Oct 30 21:02:24.265492 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:24.265255    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnszs\" (UniqueName: \"kubernetes.io/projected/7bd27e7e-be97-4796-84dd-72f42b5bf42e-kube-api-access-hnszs\") pod \"maas-api-revoke-sub-214c7679-bnjxj\" (UID: \"7bd27e7e-be97-4796-84dd-72f42b5bf42e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-214c7679-bnjxj"
Oct 30 21:02:24.265492 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:24.265316    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bd27e7e-be97-4796-84dd-72f42b5bf42e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-214c7679-bnjxj\" (UID: \"7bd27e7e-be97-4796-84dd-72f42b5bf42e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-214c7679-bnjxj"
Oct 30 21:02:24.265724 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:24.265520    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bd27e7e-be97-4796-84dd-72f42b5bf42e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-214c7679-bnjxj\" (UID: \"7bd27e7e-be97-4796-84dd-72f42b5bf42e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-214c7679-bnjxj"
Oct 30 21:02:24.367446 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:24.367310    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnszs\" (UniqueName: \"kubernetes.io/projected/7bd27e7e-be97-4796-84dd-72f42b5bf42e-kube-api-access-hnszs\") pod \"maas-api-revoke-sub-214c7679-bnjxj\" (UID: \"7bd27e7e-be97-4796-84dd-72f42b5bf42e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-214c7679-bnjxj"
Oct 30 21:02:24.367446 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:24.367426    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bd27e7e-be97-4796-84dd-72f42b5bf42e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-214c7679-bnjxj\" (UID: \"7bd27e7e-be97-4796-84dd-72f42b5bf42e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-214c7679-bnjxj"
Oct 30 21:02:24.367646 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:24.367475    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bd27e7e-be97-4796-84dd-72f42b5bf42e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-214c7679-bnjxj\" (UID: \"7bd27e7e-be97-4796-84dd-72f42b5bf42e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-214c7679-bnjxj"
Oct 30 21:02:24.368208 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:24.368184    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bd27e7e-be97-4796-84dd-72f42b5bf42e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-214c7679-bnjxj\" (UID: \"7bd27e7e-be97-4796-84dd-72f42b5bf42e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-214c7679-bnjxj"
Oct 30 21:02:24.370250 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:24.370225    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bd27e7e-be97-4796-84dd-72f42b5bf42e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-214c7679-bnjxj\" (UID: \"7bd27e7e-be97-4796-84dd-72f42b5bf42e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-214c7679-bnjxj"
Oct 30 21:02:24.380455 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:24.380428    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnszs\" (UniqueName: \"kubernetes.io/projected/7bd27e7e-be97-4796-84dd-72f42b5bf42e-kube-api-access-hnszs\") pod \"maas-api-revoke-sub-214c7679-bnjxj\" (UID: \"7bd27e7e-be97-4796-84dd-72f42b5bf42e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-214c7679-bnjxj"
Oct 30 21:02:24.399366 ip-10-0-131-125 kubenswrapper[4521]: W1030 21:02:24.399320    4521 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28e05ac8_5b4d_4085_bf8c_58793ca1c370.slice/crio-09e87986b19588b8c07b2b40e329dc86bab43627676825e0faec8ef397dc7f9d WatchSource:0}: Error finding container 09e87986b19588b8c07b2b40e329dc86bab43627676825e0faec8ef397dc7f9d: Status 404 returned error can't find the container with id 09e87986b19588b8c07b2b40e329dc86bab43627676825e0faec8ef397dc7f9d
Oct 30 21:02:24.400062 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:24.400035    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c2d1eb-9m5xb"]
Oct 30 21:02:24.504109 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:24.504069    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-214c7679-bnjxj"
Oct 30 21:02:24.508896 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:24.508871    4521 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6f6a826-ggh4m"]
Oct 30 21:02:24.515188 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:24.515159    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6f6a826-ggh4m"
Oct 30 21:02:24.539964 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:24.539936    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6f6a826-ggh4m"]
Oct 30 21:02:24.680981 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:24.675179    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfwlq\" (UniqueName: \"kubernetes.io/projected/9b9a4c83-9045-460b-bec6-8c2a6c3615a3-kube-api-access-sfwlq\") pod \"maas-api-revoke-sub-d6f6a826-ggh4m\" (UID: \"9b9a4c83-9045-460b-bec6-8c2a6c3615a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6f6a826-ggh4m"
Oct 30 21:02:24.680981 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:24.675227    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b9a4c83-9045-460b-bec6-8c2a6c3615a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d6f6a826-ggh4m\" (UID: \"9b9a4c83-9045-460b-bec6-8c2a6c3615a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6f6a826-ggh4m"
Oct 30 21:02:24.680981 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:24.675481    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b9a4c83-9045-460b-bec6-8c2a6c3615a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d6f6a826-ggh4m\" (UID: \"9b9a4c83-9045-460b-bec6-8c2a6c3615a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6f6a826-ggh4m"
Oct 30 21:02:24.767785 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:24.767742    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c2d1eb-9m5xb" event={"ID":"28e05ac8-5b4d-4085-bf8c-58793ca1c370","Type":"ContainerStarted","Data":"09e87986b19588b8c07b2b40e329dc86bab43627676825e0faec8ef397dc7f9d"}
Oct 30 21:02:24.776520 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:24.776492    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b9a4c83-9045-460b-bec6-8c2a6c3615a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d6f6a826-ggh4m\" (UID: \"9b9a4c83-9045-460b-bec6-8c2a6c3615a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6f6a826-ggh4m"
Oct 30 21:02:24.776653 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:24.776594    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfwlq\" (UniqueName: \"kubernetes.io/projected/9b9a4c83-9045-460b-bec6-8c2a6c3615a3-kube-api-access-sfwlq\") pod \"maas-api-revoke-sub-d6f6a826-ggh4m\" (UID: \"9b9a4c83-9045-460b-bec6-8c2a6c3615a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6f6a826-ggh4m"
Oct 30 21:02:24.776653 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:24.776627    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b9a4c83-9045-460b-bec6-8c2a6c3615a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d6f6a826-ggh4m\" (UID: \"9b9a4c83-9045-460b-bec6-8c2a6c3615a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6f6a826-ggh4m"
Oct 30 21:02:24.777155 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:24.777137    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b9a4c83-9045-460b-bec6-8c2a6c3615a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d6f6a826-ggh4m\" (UID: \"9b9a4c83-9045-460b-bec6-8c2a6c3615a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6f6a826-ggh4m"
Oct 30 21:02:24.779019 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:24.779002    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b9a4c83-9045-460b-bec6-8c2a6c3615a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d6f6a826-ggh4m\" (UID: \"9b9a4c83-9045-460b-bec6-8c2a6c3615a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6f6a826-ggh4m"
Oct 30 21:02:24.789478 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:24.789456    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfwlq\" (UniqueName: \"kubernetes.io/projected/9b9a4c83-9045-460b-bec6-8c2a6c3615a3-kube-api-access-sfwlq\") pod \"maas-api-revoke-sub-d6f6a826-ggh4m\" (UID: \"9b9a4c83-9045-460b-bec6-8c2a6c3615a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6f6a826-ggh4m"
Oct 30 21:02:24.820353 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:24.820309    4521 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-65968b/distinct-2-w5-65968b-kserve-6d65c85f6c-dz5r7"]
Oct 30 21:02:24.837583 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:24.837537    4521 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-65968b/premium-w5-65968b-kserve-67977485f7-c9jpv"]
Oct 30 21:02:24.845534 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:24.845500    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6f6a826-ggh4m"
Oct 30 21:02:24.900653 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:24.900618    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-214c7679-bnjxj"]
Oct 30 21:02:24.901498 ip-10-0-131-125 kubenswrapper[4521]: W1030 21:02:24.901475    4521 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bd27e7e_be97_4796_84dd_72f42b5bf42e.slice/crio-50f6973008f1bd07bdb44bb23364213d93e45c7571778b752cc7de33b003c49c WatchSource:0}: Error finding container 50f6973008f1bd07bdb44bb23364213d93e45c7571778b752cc7de33b003c49c: Status 404 returned error can't find the container with id 50f6973008f1bd07bdb44bb23364213d93e45c7571778b752cc7de33b003c49c
Oct 30 21:02:25.181039 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:25.181009    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6f6a826-ggh4m"]
Oct 30 21:02:25.181742 ip-10-0-131-125 kubenswrapper[4521]: W1030 21:02:25.181710    4521 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b9a4c83_9045_460b_bec6_8c2a6c3615a3.slice/crio-50be316b11757f6a4f705635ca3f792dfc292b49e9834c2231b0862ed096c811 WatchSource:0}: Error finding container 50be316b11757f6a4f705635ca3f792dfc292b49e9834c2231b0862ed096c811: Status 404 returned error can't find the container with id 50be316b11757f6a4f705635ca3f792dfc292b49e9834c2231b0862ed096c811
Oct 30 21:02:25.494825 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:25.494795    4521 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-nvwt8"]
Oct 30 21:02:25.498880 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:25.498849    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-nvwt8"
Oct 30 21:02:25.515805 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:25.515778    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-nvwt8"]
Oct 30 21:02:25.551069 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:25.549087    4521 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-m9wn4"]
Oct 30 21:02:25.558596 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:25.558570    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-m9wn4"]
Oct 30 21:02:25.568437 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:25.568411    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-m9wn4"
Oct 30 21:02:25.589355 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:25.589306    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cff2b61-dcbd-4450-b776-43eff57a4fd2-workload-socket\") pod \"e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-nvwt8\" (UID: \"6cff2b61-dcbd-4450-b776-43eff57a4fd2\") " pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-nvwt8"
Oct 30 21:02:25.589495 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:25.589457    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6cff2b61-dcbd-4450-b776-43eff57a4fd2-istio-token\") pod \"e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-nvwt8\" (UID: \"6cff2b61-dcbd-4450-b776-43eff57a4fd2\") " pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-nvwt8"
Oct 30 21:02:25.589583 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:25.589523    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6cff2b61-dcbd-4450-b776-43eff57a4fd2-istio-envoy\") pod \"e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-nvwt8\" (UID: \"6cff2b61-dcbd-4450-b776-43eff57a4fd2\") " pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-nvwt8"
Oct 30 21:02:25.589642 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:25.589614    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6cff2b61-dcbd-4450-b776-43eff57a4fd2-istio-podinfo\") pod \"e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-nvwt8\" (UID: \"6cff2b61-dcbd-4450-b776-43eff57a4fd2\") " pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-nvwt8"
Oct 30 21:02:25.589717 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:25.589691    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6cff2b61-dcbd-4450-b776-43eff57a4fd2-workload-certs\") pod \"e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-nvwt8\" (UID: \"6cff2b61-dcbd-4450-b776-43eff57a4fd2\") " pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-nvwt8"
Oct 30 21:02:25.589884 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:25.589860    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cff2b61-dcbd-4450-b776-43eff57a4fd2-credential-socket\") pod \"e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-nvwt8\" (UID: \"6cff2b61-dcbd-4450-b776-43eff57a4fd2\") " pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-nvwt8"
Oct 30 21:02:25.589961 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:25.589925    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6cff2b61-dcbd-4450-b776-43eff57a4fd2-istiod-ca-cert\") pod \"e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-nvwt8\" (UID: \"6cff2b61-dcbd-4450-b776-43eff57a4fd2\") " pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-nvwt8"
Oct 30 21:02:25.590050 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:25.590031    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgwgj\" (UniqueName: \"kubernetes.io/projected/6cff2b61-dcbd-4450-b776-43eff57a4fd2-kube-api-access-bgwgj\") pod \"e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-nvwt8\" (UID: \"6cff2b61-dcbd-4450-b776-43eff57a4fd2\") " pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-nvwt8"
Oct 30 21:02:25.590108 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:25.590068    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6cff2b61-dcbd-4450-b776-43eff57a4fd2-istio-data\") pod \"e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-nvwt8\" (UID: \"6cff2b61-dcbd-4450-b776-43eff57a4fd2\") " pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-nvwt8"
Oct 30 21:02:25.691401 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:25.691363    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6cff2b61-dcbd-4450-b776-43eff57a4fd2-istiod-ca-cert\") pod \"e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-nvwt8\" (UID: \"6cff2b61-dcbd-4450-b776-43eff57a4fd2\") " pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-nvwt8"
Oct 30 21:02:25.691587 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:25.691424    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgwgj\" (UniqueName: \"kubernetes.io/projected/6cff2b61-dcbd-4450-b776-43eff57a4fd2-kube-api-access-bgwgj\") pod \"e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-nvwt8\" (UID: \"6cff2b61-dcbd-4450-b776-43eff57a4fd2\") " pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-nvwt8"
Oct 30 21:02:25.691587 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:25.691447    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6cff2b61-dcbd-4450-b776-43eff57a4fd2-istio-data\") pod \"e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-nvwt8\" (UID: \"6cff2b61-dcbd-4450-b776-43eff57a4fd2\") " pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-nvwt8"
Oct 30 21:02:25.691587 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:25.691492    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cff2b61-dcbd-4450-b776-43eff57a4fd2-workload-socket\") pod \"e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-nvwt8\" (UID: \"6cff2b61-dcbd-4450-b776-43eff57a4fd2\") " pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-nvwt8"
Oct 30 21:02:25.691587 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:25.691531    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9dc1356e-e12d-49f9-822e-f1ccf4fdaa3b-istiod-ca-cert\") pod \"e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-m9wn4\" (UID: \"9dc1356e-e12d-49f9-822e-f1ccf4fdaa3b\") " pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-m9wn4"
Oct 30 21:02:25.691587 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:25.691571    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9dc1356e-e12d-49f9-822e-f1ccf4fdaa3b-istio-data\") pod \"e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-m9wn4\" (UID: \"9dc1356e-e12d-49f9-822e-f1ccf4fdaa3b\") " pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-m9wn4"
Oct 30 21:02:25.691873 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:25.691673    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cc4j\" (UniqueName: \"kubernetes.io/projected/9dc1356e-e12d-49f9-822e-f1ccf4fdaa3b-kube-api-access-4cc4j\") pod \"e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-m9wn4\" (UID: \"9dc1356e-e12d-49f9-822e-f1ccf4fdaa3b\") " pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-m9wn4"
Oct 30 21:02:25.691873 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:25.691735    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6cff2b61-dcbd-4450-b776-43eff57a4fd2-istio-token\") pod \"e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-nvwt8\" (UID: \"6cff2b61-dcbd-4450-b776-43eff57a4fd2\") " pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-nvwt8"
Oct 30 21:02:25.691873 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:25.691796    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dc1356e-e12d-49f9-822e-f1ccf4fdaa3b-credential-socket\") pod \"e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-m9wn4\" (UID: \"9dc1356e-e12d-49f9-822e-f1ccf4fdaa3b\") " pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-m9wn4"
Oct 30 21:02:25.691873 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:25.691830    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9dc1356e-e12d-49f9-822e-f1ccf4fdaa3b-istio-token\") pod \"e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-m9wn4\" (UID: \"9dc1356e-e12d-49f9-822e-f1ccf4fdaa3b\") " pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-m9wn4"
Oct 30 21:02:25.691873 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:25.691870    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6cff2b61-dcbd-4450-b776-43eff57a4fd2-istio-envoy\") pod \"e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-nvwt8\" (UID: \"6cff2b61-dcbd-4450-b776-43eff57a4fd2\") " pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-nvwt8"
Oct 30 21:02:25.692111 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:25.691910    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6cff2b61-dcbd-4450-b776-43eff57a4fd2-istio-data\") pod \"e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-nvwt8\" (UID: \"6cff2b61-dcbd-4450-b776-43eff57a4fd2\") " pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-nvwt8"
Oct 30 21:02:25.692111 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:25.691939    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cff2b61-dcbd-4450-b776-43eff57a4fd2-workload-socket\") pod \"e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-nvwt8\" (UID: \"6cff2b61-dcbd-4450-b776-43eff57a4fd2\") " pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-nvwt8"
Oct 30 21:02:25.692111 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:25.691957    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9dc1356e-e12d-49f9-822e-f1ccf4fdaa3b-workload-certs\") pod \"e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-m9wn4\" (UID: \"9dc1356e-e12d-49f9-822e-f1ccf4fdaa3b\") " pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-m9wn4"
Oct 30 21:02:25.692111 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:25.692015    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6cff2b61-dcbd-4450-b776-43eff57a4fd2-istio-podinfo\") pod \"e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-nvwt8\" (UID: \"6cff2b61-dcbd-4450-b776-43eff57a4fd2\") " pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-nvwt8"
Oct 30 21:02:25.692111 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:25.692096    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9dc1356e-e12d-49f9-822e-f1ccf4fdaa3b-istio-podinfo\") pod \"e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-m9wn4\" (UID: \"9dc1356e-e12d-49f9-822e-f1ccf4fdaa3b\") " pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-m9wn4"
Oct 30 21:02:25.692401 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:25.692128    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6cff2b61-dcbd-4450-b776-43eff57a4fd2-istiod-ca-cert\") pod \"e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-nvwt8\" (UID: \"6cff2b61-dcbd-4450-b776-43eff57a4fd2\") " pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-nvwt8"
Oct 30 21:02:25.692401 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:25.692194    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6cff2b61-dcbd-4450-b776-43eff57a4fd2-workload-certs\") pod \"e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-nvwt8\" (UID: \"6cff2b61-dcbd-4450-b776-43eff57a4fd2\") " pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-nvwt8"
Oct 30 21:02:25.692401 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:25.692240    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cff2b61-dcbd-4450-b776-43eff57a4fd2-credential-socket\") pod \"e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-nvwt8\" (UID: \"6cff2b61-dcbd-4450-b776-43eff57a4fd2\") " pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-nvwt8"
Oct 30 21:02:25.692401 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:25.692261    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dc1356e-e12d-49f9-822e-f1ccf4fdaa3b-workload-socket\") pod \"e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-m9wn4\" (UID: \"9dc1356e-e12d-49f9-822e-f1ccf4fdaa3b\") " pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-m9wn4"
Oct 30 21:02:25.692401 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:25.692290    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9dc1356e-e12d-49f9-822e-f1ccf4fdaa3b-istio-envoy\") pod \"e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-m9wn4\" (UID: \"9dc1356e-e12d-49f9-822e-f1ccf4fdaa3b\") " pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-m9wn4"
Oct 30 21:02:25.692604 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:25.692488    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6cff2b61-dcbd-4450-b776-43eff57a4fd2-workload-certs\") pod \"e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-nvwt8\" (UID: \"6cff2b61-dcbd-4450-b776-43eff57a4fd2\") " pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-nvwt8"
Oct 30 21:02:25.692655 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:25.692610    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cff2b61-dcbd-4450-b776-43eff57a4fd2-credential-socket\") pod \"e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-nvwt8\" (UID: \"6cff2b61-dcbd-4450-b776-43eff57a4fd2\") " pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-nvwt8"
Oct 30 21:02:25.694211 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:25.694181    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6cff2b61-dcbd-4450-b776-43eff57a4fd2-istio-envoy\") pod \"e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-nvwt8\" (UID: \"6cff2b61-dcbd-4450-b776-43eff57a4fd2\") " pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-nvwt8"
Oct 30 21:02:25.694553 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:25.694533    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6cff2b61-dcbd-4450-b776-43eff57a4fd2-istio-podinfo\") pod \"e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-nvwt8\" (UID: \"6cff2b61-dcbd-4450-b776-43eff57a4fd2\") " pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-nvwt8"
Oct 30 21:02:25.704060 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:25.704007    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6cff2b61-dcbd-4450-b776-43eff57a4fd2-istio-token\") pod \"e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-nvwt8\" (UID: \"6cff2b61-dcbd-4450-b776-43eff57a4fd2\") " pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-nvwt8"
Oct 30 21:02:25.704378 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:25.704357    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgwgj\" (UniqueName: \"kubernetes.io/projected/6cff2b61-dcbd-4450-b776-43eff57a4fd2-kube-api-access-bgwgj\") pod \"e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-nvwt8\" (UID: \"6cff2b61-dcbd-4450-b776-43eff57a4fd2\") " pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-nvwt8"
Oct 30 21:02:25.777040 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:25.777003    4521 generic.go:356] "Generic (PLEG): container finished" podID="28e05ac8-5b4d-4085-bf8c-58793ca1c370" containerID="99129f8e7124d04a27ac49c5fe925c7dbcb4e81dca8a1358e7d504d0b18a897d" exitCode=0
Oct 30 21:02:25.777524 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:25.777088    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c2d1eb-9m5xb" event={"ID":"28e05ac8-5b4d-4085-bf8c-58793ca1c370","Type":"ContainerDied","Data":"99129f8e7124d04a27ac49c5fe925c7dbcb4e81dca8a1358e7d504d0b18a897d"}
Oct 30 21:02:25.778829 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:25.778798    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-214c7679-bnjxj" event={"ID":"7bd27e7e-be97-4796-84dd-72f42b5bf42e","Type":"ContainerStarted","Data":"47e96b2aa0754a22e6b855b58362b8058daa069b9da3e86fce6eda12764f3c7d"}
Oct 30 21:02:25.778956 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:25.778848    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-214c7679-bnjxj" event={"ID":"7bd27e7e-be97-4796-84dd-72f42b5bf42e","Type":"ContainerStarted","Data":"50f6973008f1bd07bdb44bb23364213d93e45c7571778b752cc7de33b003c49c"}
Oct 30 21:02:25.780375 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:25.780351    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6f6a826-ggh4m" event={"ID":"9b9a4c83-9045-460b-bec6-8c2a6c3615a3","Type":"ContainerStarted","Data":"2902133771aa127fda482d8b7422a5e0b96f65de64b58535eae12ed27c79d77a"}
Oct 30 21:02:25.780453 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:25.780396    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6f6a826-ggh4m" event={"ID":"9b9a4c83-9045-460b-bec6-8c2a6c3615a3","Type":"ContainerStarted","Data":"50be316b11757f6a4f705635ca3f792dfc292b49e9834c2231b0862ed096c811"}
Oct 30 21:02:25.793625 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:25.793594    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9dc1356e-e12d-49f9-822e-f1ccf4fdaa3b-workload-certs\") pod \"e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-m9wn4\" (UID: \"9dc1356e-e12d-49f9-822e-f1ccf4fdaa3b\") " pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-m9wn4"
Oct 30 21:02:25.793719 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:25.793674    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9dc1356e-e12d-49f9-822e-f1ccf4fdaa3b-istio-podinfo\") pod \"e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-m9wn4\" (UID: \"9dc1356e-e12d-49f9-822e-f1ccf4fdaa3b\") " pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-m9wn4"
Oct 30 21:02:25.793762 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:25.793744    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dc1356e-e12d-49f9-822e-f1ccf4fdaa3b-workload-socket\") pod \"e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-m9wn4\" (UID: \"9dc1356e-e12d-49f9-822e-f1ccf4fdaa3b\") " pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-m9wn4"
Oct 30 21:02:25.793818 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:25.793787    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9dc1356e-e12d-49f9-822e-f1ccf4fdaa3b-istio-envoy\") pod \"e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-m9wn4\" (UID: \"9dc1356e-e12d-49f9-822e-f1ccf4fdaa3b\") " pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-m9wn4"
Oct 30 21:02:25.793883 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:25.793871    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9dc1356e-e12d-49f9-822e-f1ccf4fdaa3b-istiod-ca-cert\") pod \"e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-m9wn4\" (UID: \"9dc1356e-e12d-49f9-822e-f1ccf4fdaa3b\") " pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-m9wn4"
Oct 30 21:02:25.793920 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:25.793905    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9dc1356e-e12d-49f9-822e-f1ccf4fdaa3b-istio-data\") pod \"e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-m9wn4\" (UID: \"9dc1356e-e12d-49f9-822e-f1ccf4fdaa3b\") " pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-m9wn4"
Oct 30 21:02:25.793964 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:25.793934    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cc4j\" (UniqueName: \"kubernetes.io/projected/9dc1356e-e12d-49f9-822e-f1ccf4fdaa3b-kube-api-access-4cc4j\") pod \"e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-m9wn4\" (UID: \"9dc1356e-e12d-49f9-822e-f1ccf4fdaa3b\") " pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-m9wn4"
Oct 30 21:02:25.794012 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:25.793991    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dc1356e-e12d-49f9-822e-f1ccf4fdaa3b-credential-socket\") pod \"e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-m9wn4\" (UID: \"9dc1356e-e12d-49f9-822e-f1ccf4fdaa3b\") " pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-m9wn4"
Oct 30 21:02:25.794066 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:25.794018    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9dc1356e-e12d-49f9-822e-f1ccf4fdaa3b-istio-token\") pod \"e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-m9wn4\" (UID: \"9dc1356e-e12d-49f9-822e-f1ccf4fdaa3b\") " pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-m9wn4"
Oct 30 21:02:25.794066 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:25.794029    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9dc1356e-e12d-49f9-822e-f1ccf4fdaa3b-workload-certs\") pod \"e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-m9wn4\" (UID: \"9dc1356e-e12d-49f9-822e-f1ccf4fdaa3b\") " pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-m9wn4"
Oct 30 21:02:25.794167 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:25.794114    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dc1356e-e12d-49f9-822e-f1ccf4fdaa3b-workload-socket\") pod \"e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-m9wn4\" (UID: \"9dc1356e-e12d-49f9-822e-f1ccf4fdaa3b\") " pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-m9wn4"
Oct 30 21:02:25.794218 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:25.794205    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9dc1356e-e12d-49f9-822e-f1ccf4fdaa3b-istio-data\") pod \"e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-m9wn4\" (UID: \"9dc1356e-e12d-49f9-822e-f1ccf4fdaa3b\") " pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-m9wn4"
Oct 30 21:02:25.794386 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:25.794365    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dc1356e-e12d-49f9-822e-f1ccf4fdaa3b-credential-socket\") pod \"e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-m9wn4\" (UID: \"9dc1356e-e12d-49f9-822e-f1ccf4fdaa3b\") " pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-m9wn4"
Oct 30 21:02:25.794780 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:25.794753    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9dc1356e-e12d-49f9-822e-f1ccf4fdaa3b-istiod-ca-cert\") pod \"e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-m9wn4\" (UID: \"9dc1356e-e12d-49f9-822e-f1ccf4fdaa3b\") " pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-m9wn4"
Oct 30 21:02:25.796498 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:25.796478    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9dc1356e-e12d-49f9-822e-f1ccf4fdaa3b-istio-podinfo\") pod \"e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-m9wn4\" (UID: \"9dc1356e-e12d-49f9-822e-f1ccf4fdaa3b\") " pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-m9wn4"
Oct 30 21:02:25.796762 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:25.796745    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9dc1356e-e12d-49f9-822e-f1ccf4fdaa3b-istio-envoy\") pod \"e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-m9wn4\" (UID: \"9dc1356e-e12d-49f9-822e-f1ccf4fdaa3b\") " pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-m9wn4"
Oct 30 21:02:25.819197 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:25.819162    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cc4j\" (UniqueName: \"kubernetes.io/projected/9dc1356e-e12d-49f9-822e-f1ccf4fdaa3b-kube-api-access-4cc4j\") pod \"e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-m9wn4\" (UID: \"9dc1356e-e12d-49f9-822e-f1ccf4fdaa3b\") " pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-m9wn4"
Oct 30 21:02:25.819750 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:25.819703    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9dc1356e-e12d-49f9-822e-f1ccf4fdaa3b-istio-token\") pod \"e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-m9wn4\" (UID: \"9dc1356e-e12d-49f9-822e-f1ccf4fdaa3b\") " pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-m9wn4"
Oct 30 21:02:25.820451 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:25.820396    4521 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6f6a826-ggh4m" podStartSLOduration=1.8203796319999999 podStartE2EDuration="1.820379632s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:02:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:02:25.816372966 +0000 UTC m=+1413.161994202" watchObservedRunningTime="2026-10-30 21:02:25.820379632 +0000 UTC m=+1413.166000881"
Oct 30 21:02:25.824682 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:25.824653    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-nvwt8"
Oct 30 21:02:25.841717 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:25.841661    4521 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-214c7679-bnjxj" podStartSLOduration=1.8416394550000001 podStartE2EDuration="1.841639455s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:02:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:02:25.835797042 +0000 UTC m=+1413.181418282" watchObservedRunningTime="2026-10-30 21:02:25.841639455 +0000 UTC m=+1413.187260694"
Oct 30 21:02:25.900348 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:25.898730    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-m9wn4"
Oct 30 21:02:26.137747 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:26.136910    4521 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:02:26.137747 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:26.137002    4521 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:02:26.137747 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:26.137047    4521 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:02:26.145594 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:26.145444    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-nvwt8"]
Oct 30 21:02:26.202041 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:26.202007    4521 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-39fb96-openshift-default-5f55489475-z6jxt"]
Oct 30 21:02:26.214533 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:26.213709    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-39fb96-openshift-default-5f55489475-z6jxt"
Oct 30 21:02:26.226907 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:26.226871    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-39fb96-openshift-default-5f55489475-z6jxt"]
Oct 30 21:02:26.272418 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:26.272386    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-m9wn4"]
Oct 30 21:02:26.279392 ip-10-0-131-125 kubenswrapper[4521]: W1030 21:02:26.276545    4521 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9dc1356e_e12d_49f9_822e_f1ccf4fdaa3b.slice/crio-9530665c69204811e9a52e6b2ac4dbc591bb52a655d56f2dfe0be2f7bf3b5cf1 WatchSource:0}: Error finding container 9530665c69204811e9a52e6b2ac4dbc591bb52a655d56f2dfe0be2f7bf3b5cf1: Status 404 returned error can't find the container with id 9530665c69204811e9a52e6b2ac4dbc591bb52a655d56f2dfe0be2f7bf3b5cf1
Oct 30 21:02:26.279392 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:26.279267    4521 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:02:26.279619 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:26.279397    4521 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:02:26.279619 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:26.279434    4521 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:02:26.284253 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:26.284222    4521 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-9zp68"]
Oct 30 21:02:26.291975 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:26.291951    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-9zp68"
Oct 30 21:02:26.307503 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:26.305373    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3990d1c-906f-4267-b67b-5b1309f20cfc-workload-socket\") pod \"e2e-worker-w6-39fb96-openshift-default-5f55489475-z6jxt\" (UID: \"d3990d1c-906f-4267-b67b-5b1309f20cfc\") " pod="openshift-ingress/e2e-worker-w6-39fb96-openshift-default-5f55489475-z6jxt"
Oct 30 21:02:26.307503 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:26.305431    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3990d1c-906f-4267-b67b-5b1309f20cfc-istio-envoy\") pod \"e2e-worker-w6-39fb96-openshift-default-5f55489475-z6jxt\" (UID: \"d3990d1c-906f-4267-b67b-5b1309f20cfc\") " pod="openshift-ingress/e2e-worker-w6-39fb96-openshift-default-5f55489475-z6jxt"
Oct 30 21:02:26.307503 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:26.305527    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3990d1c-906f-4267-b67b-5b1309f20cfc-istio-podinfo\") pod \"e2e-worker-w6-39fb96-openshift-default-5f55489475-z6jxt\" (UID: \"d3990d1c-906f-4267-b67b-5b1309f20cfc\") " pod="openshift-ingress/e2e-worker-w6-39fb96-openshift-default-5f55489475-z6jxt"
Oct 30 21:02:26.307503 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:26.305557    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3990d1c-906f-4267-b67b-5b1309f20cfc-istiod-ca-cert\") pod \"e2e-worker-w6-39fb96-openshift-default-5f55489475-z6jxt\" (UID: \"d3990d1c-906f-4267-b67b-5b1309f20cfc\") " pod="openshift-ingress/e2e-worker-w6-39fb96-openshift-default-5f55489475-z6jxt"
Oct 30 21:02:26.307503 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:26.305591    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dfzh\" (UniqueName: \"kubernetes.io/projected/d3990d1c-906f-4267-b67b-5b1309f20cfc-kube-api-access-7dfzh\") pod \"e2e-worker-w6-39fb96-openshift-default-5f55489475-z6jxt\" (UID: \"d3990d1c-906f-4267-b67b-5b1309f20cfc\") " pod="openshift-ingress/e2e-worker-w6-39fb96-openshift-default-5f55489475-z6jxt"
Oct 30 21:02:26.307503 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:26.305637    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3990d1c-906f-4267-b67b-5b1309f20cfc-workload-certs\") pod \"e2e-worker-w6-39fb96-openshift-default-5f55489475-z6jxt\" (UID: \"d3990d1c-906f-4267-b67b-5b1309f20cfc\") " pod="openshift-ingress/e2e-worker-w6-39fb96-openshift-default-5f55489475-z6jxt"
Oct 30 21:02:26.345555 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:26.308450    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3990d1c-906f-4267-b67b-5b1309f20cfc-credential-socket\") pod \"e2e-worker-w6-39fb96-openshift-default-5f55489475-z6jxt\" (UID: \"d3990d1c-906f-4267-b67b-5b1309f20cfc\") " pod="openshift-ingress/e2e-worker-w6-39fb96-openshift-default-5f55489475-z6jxt"
Oct 30 21:02:26.345555 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:26.308519    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3990d1c-906f-4267-b67b-5b1309f20cfc-istio-data\") pod \"e2e-worker-w6-39fb96-openshift-default-5f55489475-z6jxt\" (UID: \"d3990d1c-906f-4267-b67b-5b1309f20cfc\") " pod="openshift-ingress/e2e-worker-w6-39fb96-openshift-default-5f55489475-z6jxt"
Oct 30 21:02:26.345555 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:26.308581    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3990d1c-906f-4267-b67b-5b1309f20cfc-istio-token\") pod \"e2e-worker-w6-39fb96-openshift-default-5f55489475-z6jxt\" (UID: \"d3990d1c-906f-4267-b67b-5b1309f20cfc\") " pod="openshift-ingress/e2e-worker-w6-39fb96-openshift-default-5f55489475-z6jxt"
Oct 30 21:02:26.395433 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:26.395382    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-9zp68"]
Oct 30 21:02:26.402516 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:26.402481    4521 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-sbpfk"]
Oct 30 21:02:26.412567 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:26.408320    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-sbpfk"
Oct 30 21:02:26.416056 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:26.416024    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65e4dd7b-506f-4350-a82c-e35e61706595-istio-data\") pod \"e2e-worker-w5-65968b-openshift-default-68896fcddc-9zp68\" (UID: \"65e4dd7b-506f-4350-a82c-e35e61706595\") " pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-9zp68"
Oct 30 21:02:26.416187 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:26.416138    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dfzh\" (UniqueName: \"kubernetes.io/projected/d3990d1c-906f-4267-b67b-5b1309f20cfc-kube-api-access-7dfzh\") pod \"e2e-worker-w6-39fb96-openshift-default-5f55489475-z6jxt\" (UID: \"d3990d1c-906f-4267-b67b-5b1309f20cfc\") " pod="openshift-ingress/e2e-worker-w6-39fb96-openshift-default-5f55489475-z6jxt"
Oct 30 21:02:26.416187 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:26.416174    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65e4dd7b-506f-4350-a82c-e35e61706595-istiod-ca-cert\") pod \"e2e-worker-w5-65968b-openshift-default-68896fcddc-9zp68\" (UID: \"65e4dd7b-506f-4350-a82c-e35e61706595\") " pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-9zp68"
Oct 30 21:02:26.416353 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:26.416199    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mm9tf\" (UniqueName: \"kubernetes.io/projected/65e4dd7b-506f-4350-a82c-e35e61706595-kube-api-access-mm9tf\") pod \"e2e-worker-w5-65968b-openshift-default-68896fcddc-9zp68\" (UID: \"65e4dd7b-506f-4350-a82c-e35e61706595\") " pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-9zp68"
Oct 30 21:02:26.416353 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:26.416305    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65e4dd7b-506f-4350-a82c-e35e61706595-istio-token\") pod \"e2e-worker-w5-65968b-openshift-default-68896fcddc-9zp68\" (UID: \"65e4dd7b-506f-4350-a82c-e35e61706595\") " pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-9zp68"
Oct 30 21:02:26.417009 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:26.416354    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3990d1c-906f-4267-b67b-5b1309f20cfc-istio-data\") pod \"e2e-worker-w6-39fb96-openshift-default-5f55489475-z6jxt\" (UID: \"d3990d1c-906f-4267-b67b-5b1309f20cfc\") " pod="openshift-ingress/e2e-worker-w6-39fb96-openshift-default-5f55489475-z6jxt"
Oct 30 21:02:26.417009 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:26.416515    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3990d1c-906f-4267-b67b-5b1309f20cfc-istio-token\") pod \"e2e-worker-w6-39fb96-openshift-default-5f55489475-z6jxt\" (UID: \"d3990d1c-906f-4267-b67b-5b1309f20cfc\") " pod="openshift-ingress/e2e-worker-w6-39fb96-openshift-default-5f55489475-z6jxt"
Oct 30 21:02:26.417009 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:26.416571    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3990d1c-906f-4267-b67b-5b1309f20cfc-workload-socket\") pod \"e2e-worker-w6-39fb96-openshift-default-5f55489475-z6jxt\" (UID: \"d3990d1c-906f-4267-b67b-5b1309f20cfc\") " pod="openshift-ingress/e2e-worker-w6-39fb96-openshift-default-5f55489475-z6jxt"
Oct 30 21:02:26.417009 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:26.416635    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3990d1c-906f-4267-b67b-5b1309f20cfc-istio-envoy\") pod \"e2e-worker-w6-39fb96-openshift-default-5f55489475-z6jxt\" (UID: \"d3990d1c-906f-4267-b67b-5b1309f20cfc\") " pod="openshift-ingress/e2e-worker-w6-39fb96-openshift-default-5f55489475-z6jxt"
Oct 30 21:02:26.417009 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:26.416683    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3990d1c-906f-4267-b67b-5b1309f20cfc-istio-podinfo\") pod \"e2e-worker-w6-39fb96-openshift-default-5f55489475-z6jxt\" (UID: \"d3990d1c-906f-4267-b67b-5b1309f20cfc\") " pod="openshift-ingress/e2e-worker-w6-39fb96-openshift-default-5f55489475-z6jxt"
Oct 30 21:02:26.417009 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:26.416710    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3990d1c-906f-4267-b67b-5b1309f20cfc-istiod-ca-cert\") pod \"e2e-worker-w6-39fb96-openshift-default-5f55489475-z6jxt\" (UID: \"d3990d1c-906f-4267-b67b-5b1309f20cfc\") " pod="openshift-ingress/e2e-worker-w6-39fb96-openshift-default-5f55489475-z6jxt"
Oct 30 21:02:26.417009 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:26.416749    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65e4dd7b-506f-4350-a82c-e35e61706595-workload-socket\") pod \"e2e-worker-w5-65968b-openshift-default-68896fcddc-9zp68\" (UID: \"65e4dd7b-506f-4350-a82c-e35e61706595\") " pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-9zp68"
Oct 30 21:02:26.417009 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:26.416789    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3990d1c-906f-4267-b67b-5b1309f20cfc-workload-certs\") pod \"e2e-worker-w6-39fb96-openshift-default-5f55489475-z6jxt\" (UID: \"d3990d1c-906f-4267-b67b-5b1309f20cfc\") " pod="openshift-ingress/e2e-worker-w6-39fb96-openshift-default-5f55489475-z6jxt"
Oct 30 21:02:26.417009 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:26.416820    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65e4dd7b-506f-4350-a82c-e35e61706595-workload-certs\") pod \"e2e-worker-w5-65968b-openshift-default-68896fcddc-9zp68\" (UID: \"65e4dd7b-506f-4350-a82c-e35e61706595\") " pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-9zp68"
Oct 30 21:02:26.417009 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:26.416851    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65e4dd7b-506f-4350-a82c-e35e61706595-credential-socket\") pod \"e2e-worker-w5-65968b-openshift-default-68896fcddc-9zp68\" (UID: \"65e4dd7b-506f-4350-a82c-e35e61706595\") " pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-9zp68"
Oct 30 21:02:26.417009 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:26.416880    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3990d1c-906f-4267-b67b-5b1309f20cfc-credential-socket\") pod \"e2e-worker-w6-39fb96-openshift-default-5f55489475-z6jxt\" (UID: \"d3990d1c-906f-4267-b67b-5b1309f20cfc\") " pod="openshift-ingress/e2e-worker-w6-39fb96-openshift-default-5f55489475-z6jxt"
Oct 30 21:02:26.417009 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:26.416918    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65e4dd7b-506f-4350-a82c-e35e61706595-istio-envoy\") pod \"e2e-worker-w5-65968b-openshift-default-68896fcddc-9zp68\" (UID: \"65e4dd7b-506f-4350-a82c-e35e61706595\") " pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-9zp68"
Oct 30 21:02:26.417009 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:26.416962    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65e4dd7b-506f-4350-a82c-e35e61706595-istio-podinfo\") pod \"e2e-worker-w5-65968b-openshift-default-68896fcddc-9zp68\" (UID: \"65e4dd7b-506f-4350-a82c-e35e61706595\") " pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-9zp68"
Oct 30 21:02:26.417829 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:26.417803    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3990d1c-906f-4267-b67b-5b1309f20cfc-workload-socket\") pod \"e2e-worker-w6-39fb96-openshift-default-5f55489475-z6jxt\" (UID: \"d3990d1c-906f-4267-b67b-5b1309f20cfc\") " pod="openshift-ingress/e2e-worker-w6-39fb96-openshift-default-5f55489475-z6jxt"
Oct 30 21:02:26.422878 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:26.422733    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3990d1c-906f-4267-b67b-5b1309f20cfc-workload-certs\") pod \"e2e-worker-w6-39fb96-openshift-default-5f55489475-z6jxt\" (UID: \"d3990d1c-906f-4267-b67b-5b1309f20cfc\") " pod="openshift-ingress/e2e-worker-w6-39fb96-openshift-default-5f55489475-z6jxt"
Oct 30 21:02:26.422994 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:26.422944    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3990d1c-906f-4267-b67b-5b1309f20cfc-istio-envoy\") pod \"e2e-worker-w6-39fb96-openshift-default-5f55489475-z6jxt\" (UID: \"d3990d1c-906f-4267-b67b-5b1309f20cfc\") " pod="openshift-ingress/e2e-worker-w6-39fb96-openshift-default-5f55489475-z6jxt"
Oct 30 21:02:26.424210 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:26.423661    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3990d1c-906f-4267-b67b-5b1309f20cfc-credential-socket\") pod \"e2e-worker-w6-39fb96-openshift-default-5f55489475-z6jxt\" (UID: \"d3990d1c-906f-4267-b67b-5b1309f20cfc\") " pod="openshift-ingress/e2e-worker-w6-39fb96-openshift-default-5f55489475-z6jxt"
Oct 30 21:02:26.424210 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:26.424151    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3990d1c-906f-4267-b67b-5b1309f20cfc-istiod-ca-cert\") pod \"e2e-worker-w6-39fb96-openshift-default-5f55489475-z6jxt\" (UID: \"d3990d1c-906f-4267-b67b-5b1309f20cfc\") " pod="openshift-ingress/e2e-worker-w6-39fb96-openshift-default-5f55489475-z6jxt"
Oct 30 21:02:26.424773 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:26.424752    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3990d1c-906f-4267-b67b-5b1309f20cfc-istio-data\") pod \"e2e-worker-w6-39fb96-openshift-default-5f55489475-z6jxt\" (UID: \"d3990d1c-906f-4267-b67b-5b1309f20cfc\") " pod="openshift-ingress/e2e-worker-w6-39fb96-openshift-default-5f55489475-z6jxt"
Oct 30 21:02:26.426735 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:26.426712    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-sbpfk"]
Oct 30 21:02:26.430676 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:26.430622    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3990d1c-906f-4267-b67b-5b1309f20cfc-istio-token\") pod \"e2e-worker-w6-39fb96-openshift-default-5f55489475-z6jxt\" (UID: \"d3990d1c-906f-4267-b67b-5b1309f20cfc\") " pod="openshift-ingress/e2e-worker-w6-39fb96-openshift-default-5f55489475-z6jxt"
Oct 30 21:02:26.430788 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:26.430751    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3990d1c-906f-4267-b67b-5b1309f20cfc-istio-podinfo\") pod \"e2e-worker-w6-39fb96-openshift-default-5f55489475-z6jxt\" (UID: \"d3990d1c-906f-4267-b67b-5b1309f20cfc\") " pod="openshift-ingress/e2e-worker-w6-39fb96-openshift-default-5f55489475-z6jxt"
Oct 30 21:02:26.435987 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:26.435959    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dfzh\" (UniqueName: \"kubernetes.io/projected/d3990d1c-906f-4267-b67b-5b1309f20cfc-kube-api-access-7dfzh\") pod \"e2e-worker-w6-39fb96-openshift-default-5f55489475-z6jxt\" (UID: \"d3990d1c-906f-4267-b67b-5b1309f20cfc\") " pod="openshift-ingress/e2e-worker-w6-39fb96-openshift-default-5f55489475-z6jxt"
Oct 30 21:02:26.523045 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:26.519790    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65e4dd7b-506f-4350-a82c-e35e61706595-istiod-ca-cert\") pod \"e2e-worker-w5-65968b-openshift-default-68896fcddc-9zp68\" (UID: \"65e4dd7b-506f-4350-a82c-e35e61706595\") " pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-9zp68"
Oct 30 21:02:26.523045 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:26.519835    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mm9tf\" (UniqueName: \"kubernetes.io/projected/65e4dd7b-506f-4350-a82c-e35e61706595-kube-api-access-mm9tf\") pod \"e2e-worker-w5-65968b-openshift-default-68896fcddc-9zp68\" (UID: \"65e4dd7b-506f-4350-a82c-e35e61706595\") " pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-9zp68"
Oct 30 21:02:26.523045 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:26.519881    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w84kk\" (UniqueName: \"kubernetes.io/projected/7a5f91bb-7229-454f-a006-0d768f8879f4-kube-api-access-w84kk\") pod \"e2e-worker-w5-65968b-openshift-default-68896fcddc-sbpfk\" (UID: \"7a5f91bb-7229-454f-a006-0d768f8879f4\") " pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-sbpfk"
Oct 30 21:02:26.523045 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:26.519934    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a5f91bb-7229-454f-a006-0d768f8879f4-istio-data\") pod \"e2e-worker-w5-65968b-openshift-default-68896fcddc-sbpfk\" (UID: \"7a5f91bb-7229-454f-a006-0d768f8879f4\") " pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-sbpfk"
Oct 30 21:02:26.523045 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:26.519993    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a5f91bb-7229-454f-a006-0d768f8879f4-credential-socket\") pod \"e2e-worker-w5-65968b-openshift-default-68896fcddc-sbpfk\" (UID: \"7a5f91bb-7229-454f-a006-0d768f8879f4\") " pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-sbpfk"
Oct 30 21:02:26.523045 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:26.520020    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65e4dd7b-506f-4350-a82c-e35e61706595-istio-token\") pod \"e2e-worker-w5-65968b-openshift-default-68896fcddc-9zp68\" (UID: \"65e4dd7b-506f-4350-a82c-e35e61706595\") " pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-9zp68"
Oct 30 21:02:26.523045 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:26.520048    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a5f91bb-7229-454f-a006-0d768f8879f4-workload-socket\") pod \"e2e-worker-w5-65968b-openshift-default-68896fcddc-sbpfk\" (UID: \"7a5f91bb-7229-454f-a006-0d768f8879f4\") " pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-sbpfk"
Oct 30 21:02:26.523045 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:26.520169    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a5f91bb-7229-454f-a006-0d768f8879f4-istio-envoy\") pod \"e2e-worker-w5-65968b-openshift-default-68896fcddc-sbpfk\" (UID: \"7a5f91bb-7229-454f-a006-0d768f8879f4\") " pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-sbpfk"
Oct 30 21:02:26.523045 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:26.520210    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a5f91bb-7229-454f-a006-0d768f8879f4-workload-certs\") pod \"e2e-worker-w5-65968b-openshift-default-68896fcddc-sbpfk\" (UID: \"7a5f91bb-7229-454f-a006-0d768f8879f4\") " pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-sbpfk"
Oct 30 21:02:26.523045 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:26.520260    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65e4dd7b-506f-4350-a82c-e35e61706595-workload-socket\") pod \"e2e-worker-w5-65968b-openshift-default-68896fcddc-9zp68\" (UID: \"65e4dd7b-506f-4350-a82c-e35e61706595\") " pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-9zp68"
Oct 30 21:02:26.523045 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:26.520304    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a5f91bb-7229-454f-a006-0d768f8879f4-istio-podinfo\") pod \"e2e-worker-w5-65968b-openshift-default-68896fcddc-sbpfk\" (UID: \"7a5f91bb-7229-454f-a006-0d768f8879f4\") " pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-sbpfk"
Oct 30 21:02:26.523045 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:26.520346    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65e4dd7b-506f-4350-a82c-e35e61706595-workload-certs\") pod \"e2e-worker-w5-65968b-openshift-default-68896fcddc-9zp68\" (UID: \"65e4dd7b-506f-4350-a82c-e35e61706595\") " pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-9zp68"
Oct 30 21:02:26.523045 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:26.520379    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65e4dd7b-506f-4350-a82c-e35e61706595-credential-socket\") pod \"e2e-worker-w5-65968b-openshift-default-68896fcddc-9zp68\" (UID: \"65e4dd7b-506f-4350-a82c-e35e61706595\") " pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-9zp68"
Oct 30 21:02:26.523045 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:26.520432    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65e4dd7b-506f-4350-a82c-e35e61706595-istio-envoy\") pod \"e2e-worker-w5-65968b-openshift-default-68896fcddc-9zp68\" (UID: \"65e4dd7b-506f-4350-a82c-e35e61706595\") " pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-9zp68"
Oct 30 21:02:26.523045 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:26.520475    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65e4dd7b-506f-4350-a82c-e35e61706595-istio-podinfo\") pod \"e2e-worker-w5-65968b-openshift-default-68896fcddc-9zp68\" (UID: \"65e4dd7b-506f-4350-a82c-e35e61706595\") " pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-9zp68"
Oct 30 21:02:26.523045 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:26.520534    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65e4dd7b-506f-4350-a82c-e35e61706595-istio-data\") pod \"e2e-worker-w5-65968b-openshift-default-68896fcddc-9zp68\" (UID: \"65e4dd7b-506f-4350-a82c-e35e61706595\") " pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-9zp68"
Oct 30 21:02:26.523045 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:26.520566    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a5f91bb-7229-454f-a006-0d768f8879f4-istio-token\") pod \"e2e-worker-w5-65968b-openshift-default-68896fcddc-sbpfk\" (UID: \"7a5f91bb-7229-454f-a006-0d768f8879f4\") " pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-sbpfk"
Oct 30 21:02:26.523045 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:26.520621    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a5f91bb-7229-454f-a006-0d768f8879f4-istiod-ca-cert\") pod \"e2e-worker-w5-65968b-openshift-default-68896fcddc-sbpfk\" (UID: \"7a5f91bb-7229-454f-a006-0d768f8879f4\") " pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-sbpfk"
Oct 30 21:02:26.523045 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:26.522234    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65e4dd7b-506f-4350-a82c-e35e61706595-credential-socket\") pod \"e2e-worker-w5-65968b-openshift-default-68896fcddc-9zp68\" (UID: \"65e4dd7b-506f-4350-a82c-e35e61706595\") " pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-9zp68"
Oct 30 21:02:26.523045 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:26.522480    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65e4dd7b-506f-4350-a82c-e35e61706595-workload-socket\") pod \"e2e-worker-w5-65968b-openshift-default-68896fcddc-9zp68\" (UID: \"65e4dd7b-506f-4350-a82c-e35e61706595\") " pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-9zp68"
Oct 30 21:02:26.525105 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:26.524988    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65e4dd7b-506f-4350-a82c-e35e61706595-workload-certs\") pod \"e2e-worker-w5-65968b-openshift-default-68896fcddc-9zp68\" (UID: \"65e4dd7b-506f-4350-a82c-e35e61706595\") " pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-9zp68"
Oct 30 21:02:26.525875 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:26.525601    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65e4dd7b-506f-4350-a82c-e35e61706595-istio-data\") pod \"e2e-worker-w5-65968b-openshift-default-68896fcddc-9zp68\" (UID: \"65e4dd7b-506f-4350-a82c-e35e61706595\") " pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-9zp68"
Oct 30 21:02:26.530185 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:26.530082    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65e4dd7b-506f-4350-a82c-e35e61706595-istiod-ca-cert\") pod \"e2e-worker-w5-65968b-openshift-default-68896fcddc-9zp68\" (UID: \"65e4dd7b-506f-4350-a82c-e35e61706595\") " pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-9zp68"
Oct 30 21:02:26.556589 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:26.556380    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-39fb96-openshift-default-5f55489475-z6jxt"
Oct 30 21:02:26.556939 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:26.556915    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65e4dd7b-506f-4350-a82c-e35e61706595-istio-podinfo\") pod \"e2e-worker-w5-65968b-openshift-default-68896fcddc-9zp68\" (UID: \"65e4dd7b-506f-4350-a82c-e35e61706595\") " pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-9zp68"
Oct 30 21:02:26.559702 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:26.559635    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65e4dd7b-506f-4350-a82c-e35e61706595-istio-envoy\") pod \"e2e-worker-w5-65968b-openshift-default-68896fcddc-9zp68\" (UID: \"65e4dd7b-506f-4350-a82c-e35e61706595\") " pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-9zp68"
Oct 30 21:02:26.565626 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:26.565074    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65e4dd7b-506f-4350-a82c-e35e61706595-istio-token\") pod \"e2e-worker-w5-65968b-openshift-default-68896fcddc-9zp68\" (UID: \"65e4dd7b-506f-4350-a82c-e35e61706595\") " pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-9zp68"
Oct 30 21:02:26.569574 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:26.569520    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mm9tf\" (UniqueName: \"kubernetes.io/projected/65e4dd7b-506f-4350-a82c-e35e61706595-kube-api-access-mm9tf\") pod \"e2e-worker-w5-65968b-openshift-default-68896fcddc-9zp68\" (UID: \"65e4dd7b-506f-4350-a82c-e35e61706595\") " pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-9zp68"
Oct 30 21:02:26.622400 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:26.622358    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w84kk\" (UniqueName: \"kubernetes.io/projected/7a5f91bb-7229-454f-a006-0d768f8879f4-kube-api-access-w84kk\") pod \"e2e-worker-w5-65968b-openshift-default-68896fcddc-sbpfk\" (UID: \"7a5f91bb-7229-454f-a006-0d768f8879f4\") " pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-sbpfk"
Oct 30 21:02:26.623048 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:26.623008    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a5f91bb-7229-454f-a006-0d768f8879f4-istio-data\") pod \"e2e-worker-w5-65968b-openshift-default-68896fcddc-sbpfk\" (UID: \"7a5f91bb-7229-454f-a006-0d768f8879f4\") " pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-sbpfk"
Oct 30 21:02:26.623669 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:26.623650    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a5f91bb-7229-454f-a006-0d768f8879f4-credential-socket\") pod \"e2e-worker-w5-65968b-openshift-default-68896fcddc-sbpfk\" (UID: \"7a5f91bb-7229-454f-a006-0d768f8879f4\") " pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-sbpfk"
Oct 30 21:02:26.623820 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:26.623805    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a5f91bb-7229-454f-a006-0d768f8879f4-workload-socket\") pod \"e2e-worker-w5-65968b-openshift-default-68896fcddc-sbpfk\" (UID: \"7a5f91bb-7229-454f-a006-0d768f8879f4\") " pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-sbpfk"
Oct 30 21:02:26.624035 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:26.624015    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a5f91bb-7229-454f-a006-0d768f8879f4-istio-envoy\") pod \"e2e-worker-w5-65968b-openshift-default-68896fcddc-sbpfk\" (UID: \"7a5f91bb-7229-454f-a006-0d768f8879f4\") " pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-sbpfk"
Oct 30 21:02:26.624176 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:26.624161    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a5f91bb-7229-454f-a006-0d768f8879f4-workload-certs\") pod \"e2e-worker-w5-65968b-openshift-default-68896fcddc-sbpfk\" (UID: \"7a5f91bb-7229-454f-a006-0d768f8879f4\") " pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-sbpfk"
Oct 30 21:02:26.624342 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:26.624307    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a5f91bb-7229-454f-a006-0d768f8879f4-istio-podinfo\") pod \"e2e-worker-w5-65968b-openshift-default-68896fcddc-sbpfk\" (UID: \"7a5f91bb-7229-454f-a006-0d768f8879f4\") " pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-sbpfk"
Oct 30 21:02:26.629224 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:26.624553    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a5f91bb-7229-454f-a006-0d768f8879f4-istio-token\") pod \"e2e-worker-w5-65968b-openshift-default-68896fcddc-sbpfk\" (UID: \"7a5f91bb-7229-454f-a006-0d768f8879f4\") " pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-sbpfk"
Oct 30 21:02:26.629485 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:26.629461    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a5f91bb-7229-454f-a006-0d768f8879f4-istiod-ca-cert\") pod \"e2e-worker-w5-65968b-openshift-default-68896fcddc-sbpfk\" (UID: \"7a5f91bb-7229-454f-a006-0d768f8879f4\") " pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-sbpfk"
Oct 30 21:02:26.629827 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:26.623568    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a5f91bb-7229-454f-a006-0d768f8879f4-istio-data\") pod \"e2e-worker-w5-65968b-openshift-default-68896fcddc-sbpfk\" (UID: \"7a5f91bb-7229-454f-a006-0d768f8879f4\") " pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-sbpfk"
Oct 30 21:02:26.629960 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:26.627039    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a5f91bb-7229-454f-a006-0d768f8879f4-credential-socket\") pod \"e2e-worker-w5-65968b-openshift-default-68896fcddc-sbpfk\" (UID: \"7a5f91bb-7229-454f-a006-0d768f8879f4\") " pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-sbpfk"
Oct 30 21:02:26.630048 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:26.627257    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a5f91bb-7229-454f-a006-0d768f8879f4-workload-certs\") pod \"e2e-worker-w5-65968b-openshift-default-68896fcddc-sbpfk\" (UID: \"7a5f91bb-7229-454f-a006-0d768f8879f4\") " pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-sbpfk"
Oct 30 21:02:26.630128 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:26.626826    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a5f91bb-7229-454f-a006-0d768f8879f4-workload-socket\") pod \"e2e-worker-w5-65968b-openshift-default-68896fcddc-sbpfk\" (UID: \"7a5f91bb-7229-454f-a006-0d768f8879f4\") " pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-sbpfk"
Oct 30 21:02:26.631009 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:26.630986    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a5f91bb-7229-454f-a006-0d768f8879f4-istiod-ca-cert\") pod \"e2e-worker-w5-65968b-openshift-default-68896fcddc-sbpfk\" (UID: \"7a5f91bb-7229-454f-a006-0d768f8879f4\") " pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-sbpfk"
Oct 30 21:02:26.639473 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:26.639240    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-9zp68"
Oct 30 21:02:26.717741 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:26.713737    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a5f91bb-7229-454f-a006-0d768f8879f4-istio-token\") pod \"e2e-worker-w5-65968b-openshift-default-68896fcddc-sbpfk\" (UID: \"7a5f91bb-7229-454f-a006-0d768f8879f4\") " pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-sbpfk"
Oct 30 21:02:26.727352 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:26.726353    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a5f91bb-7229-454f-a006-0d768f8879f4-istio-envoy\") pod \"e2e-worker-w5-65968b-openshift-default-68896fcddc-sbpfk\" (UID: \"7a5f91bb-7229-454f-a006-0d768f8879f4\") " pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-sbpfk"
Oct 30 21:02:26.737819 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:26.737464    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w84kk\" (UniqueName: \"kubernetes.io/projected/7a5f91bb-7229-454f-a006-0d768f8879f4-kube-api-access-w84kk\") pod \"e2e-worker-w5-65968b-openshift-default-68896fcddc-sbpfk\" (UID: \"7a5f91bb-7229-454f-a006-0d768f8879f4\") " pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-sbpfk"
Oct 30 21:02:26.750938 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:26.747912    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a5f91bb-7229-454f-a006-0d768f8879f4-istio-podinfo\") pod \"e2e-worker-w5-65968b-openshift-default-68896fcddc-sbpfk\" (UID: \"7a5f91bb-7229-454f-a006-0d768f8879f4\") " pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-sbpfk"
Oct 30 21:02:26.909000 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:26.880845    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-sbpfk"
Oct 30 21:02:26.909000 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:26.894976    4521 generic.go:356] "Generic (PLEG): container finished" podID="9b9a4c83-9045-460b-bec6-8c2a6c3615a3" containerID="2902133771aa127fda482d8b7422a5e0b96f65de64b58535eae12ed27c79d77a" exitCode=0
Oct 30 21:02:26.909000 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:26.895071    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6f6a826-ggh4m" event={"ID":"9b9a4c83-9045-460b-bec6-8c2a6c3615a3","Type":"ContainerDied","Data":"2902133771aa127fda482d8b7422a5e0b96f65de64b58535eae12ed27c79d77a"}
Oct 30 21:02:26.909000 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:26.897585    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-m9wn4" event={"ID":"9dc1356e-e12d-49f9-822e-f1ccf4fdaa3b","Type":"ContainerStarted","Data":"e420f18aec5b42561ae1b8c5b8056f63c7fbabe169afbf6204220c200e21f9aa"}
Oct 30 21:02:26.909000 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:26.897639    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-m9wn4" event={"ID":"9dc1356e-e12d-49f9-822e-f1ccf4fdaa3b","Type":"ContainerStarted","Data":"9530665c69204811e9a52e6b2ac4dbc591bb52a655d56f2dfe0be2f7bf3b5cf1"}
Oct 30 21:02:26.909000 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:26.900564    4521 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-m9wn4"
Oct 30 21:02:26.909000 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:26.901085    4521 generic.go:356] "Generic (PLEG): container finished" podID="7bd27e7e-be97-4796-84dd-72f42b5bf42e" containerID="47e96b2aa0754a22e6b855b58362b8058daa069b9da3e86fce6eda12764f3c7d" exitCode=0
Oct 30 21:02:26.909000 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:26.901209    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-214c7679-bnjxj" event={"ID":"7bd27e7e-be97-4796-84dd-72f42b5bf42e","Type":"ContainerDied","Data":"47e96b2aa0754a22e6b855b58362b8058daa069b9da3e86fce6eda12764f3c7d"}
Oct 30 21:02:26.910084 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:26.909984    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-nvwt8" event={"ID":"6cff2b61-dcbd-4450-b776-43eff57a4fd2","Type":"ContainerStarted","Data":"d0cd9f072652f1ce279084eed6d6e0420305cd803eafc75e89d9ff4abb5a09c5"}
Oct 30 21:02:26.910084 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:26.910047    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-nvwt8" event={"ID":"6cff2b61-dcbd-4450-b776-43eff57a4fd2","Type":"ContainerStarted","Data":"8985bf0af536f344755ad49bc04b93d5a08db8a67e0b9ce4571c5f0c4080ec4a"}
Oct 30 21:02:27.003111 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:27.003046    4521 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-m9wn4" podStartSLOduration=2.003019137 podStartE2EDuration="2.003019137s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:02:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:02:26.999781727 +0000 UTC m=+1414.345402966" watchObservedRunningTime="2026-10-30 21:02:27.003019137 +0000 UTC m=+1414.348640383"
Oct 30 21:02:27.108220 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:27.099904    4521 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-nvwt8" podStartSLOduration=2.099880681 podStartE2EDuration="2.099880681s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:02:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:02:27.069514204 +0000 UTC m=+1414.415135445" watchObservedRunningTime="2026-10-30 21:02:27.099880681 +0000 UTC m=+1414.445501920"
Oct 30 21:02:27.123213 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:27.122881    4521 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:02:27.123213 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:27.122983    4521 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:02:27.123213 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:27.123049    4521 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:02:27.520587 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:27.505404    4521 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:02:27.520587 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:27.505575    4521 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:02:27.520587 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:27.505671    4521 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:02:27.526396 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:27.526234    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-9zp68"]
Oct 30 21:02:27.580485 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:27.580448    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-39fb96-openshift-default-5f55489475-z6jxt"]
Oct 30 21:02:27.795646 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:27.795602    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c2d1eb-9m5xb"
Oct 30 21:02:27.838997 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:27.838958    4521 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-nvwt8"
Oct 30 21:02:27.846489 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:27.846452    4521 patch_prober.go:29] interesting pod/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-nvwt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body=
Oct 30 21:02:27.864214 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:27.864167    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-nvwt8" podUID="6cff2b61-dcbd-4450-b776-43eff57a4fd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused"
Oct 30 21:02:28.197353 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:28.192427    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28e05ac8-5b4d-4085-bf8c-58793ca1c370-kube-api-access-2ggzh\" (UniqueName: \"kubernetes.io/projected/28e05ac8-5b4d-4085-bf8c-58793ca1c370-kube-api-access-2ggzh\") pod \"28e05ac8-5b4d-4085-bf8c-58793ca1c370\" (UID: \"28e05ac8-5b4d-4085-bf8c-58793ca1c370\") "
Oct 30 21:02:28.219851 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:28.219812    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28e05ac8-5b4d-4085-bf8c-58793ca1c370-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28e05ac8-5b4d-4085-bf8c-58793ca1c370-maas-api-openshift-ca\") pod \"28e05ac8-5b4d-4085-bf8c-58793ca1c370\" (UID: \"28e05ac8-5b4d-4085-bf8c-58793ca1c370\") "
Oct 30 21:02:28.220655 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:28.220619    4521 patch_prober.go:29] interesting pod/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-m9wn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Oct 30 21:02:28.224805 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:28.224779    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28e05ac8-5b4d-4085-bf8c-58793ca1c370-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28e05ac8-5b4d-4085-bf8c-58793ca1c370-maas-api-xks-ca\") pod \"28e05ac8-5b4d-4085-bf8c-58793ca1c370\" (UID: \"28e05ac8-5b4d-4085-bf8c-58793ca1c370\") "
Oct 30 21:02:28.235627 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:28.224819    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-m9wn4" podUID="9dc1356e-e12d-49f9-822e-f1ccf4fdaa3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Oct 30 21:02:28.339349 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:28.229706    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28e05ac8-5b4d-4085-bf8c-58793ca1c370-maas-api-openshift-ca" pod "28e05ac8-5b4d-4085-bf8c-58793ca1c370" (UID: "28e05ac8-5b4d-4085-bf8c-58793ca1c370"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:02:28.339349 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:28.276502    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28e05ac8-5b4d-4085-bf8c-58793ca1c370-kube-api-access-2ggzh" pod "28e05ac8-5b4d-4085-bf8c-58793ca1c370" (UID: "28e05ac8-5b4d-4085-bf8c-58793ca1c370"). InnerVolumeSpecName "kube-api-access-2ggzh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:02:28.364347 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:28.362401    4521 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:02:28.364347 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:28.362527    4521 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:02:28.364347 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:28.362580    4521 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:02:28.364347 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:28.362998    4521 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ggzh\" (UniqueName: \"kubernetes.io/projected/28e05ac8-5b4d-4085-bf8c-58793ca1c370-kube-api-access-2ggzh\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:28.364347 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:28.363024    4521 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28e05ac8-5b4d-4085-bf8c-58793ca1c370-maas-api-openshift-ca\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:28.364347 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:28.363086    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-9zp68" event={"ID":"65e4dd7b-506f-4350-a82c-e35e61706595","Type":"ContainerStarted","Data":"ed1aa36ed724c662b14ce9b03c89f27c5184ed3f8fc9e30e3fa8989d2d8ef78f"}
Oct 30 21:02:28.364347 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:28.363144    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-9zp68" event={"ID":"65e4dd7b-506f-4350-a82c-e35e61706595","Type":"ContainerStarted","Data":"95716966944a5650b4f979d900f41ff438b9d14efdad462e0f8ccb968509e906"}
Oct 30 21:02:28.486229 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:28.486113    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c2d1eb-9m5xb" event={"ID":"28e05ac8-5b4d-4085-bf8c-58793ca1c370","Type":"ContainerDied","Data":"09e87986b19588b8c07b2b40e329dc86bab43627676825e0faec8ef397dc7f9d"}
Oct 30 21:02:28.486503 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:28.486477    4521 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="09e87986b19588b8c07b2b40e329dc86bab43627676825e0faec8ef397dc7f9d"
Oct 30 21:02:28.486738 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:28.486725    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c2d1eb-9m5xb"
Oct 30 21:02:28.534015 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:28.533974    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-39fb96-openshift-default-5f55489475-z6jxt" event={"ID":"d3990d1c-906f-4267-b67b-5b1309f20cfc","Type":"ContainerStarted","Data":"d2435b724e4f5fc6ce200c74e76d64b9c4188f66739b19ce5da0d0800f4797fa"}
Oct 30 21:02:28.555999 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:28.555957    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-39fb96-openshift-default-5f55489475-z6jxt" event={"ID":"d3990d1c-906f-4267-b67b-5b1309f20cfc","Type":"ContainerStarted","Data":"9a0fda3a0c45c18c6503f7bc0706c18db8cae2806bea478ba2d0875c3386ea5c"}
Oct 30 21:02:28.567515 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:28.567476    4521 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-39fb96-openshift-default-5f55489475-z6jxt"
Oct 30 21:02:28.594548 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:28.594499    4521 patch_prober.go:29] interesting pod/e2e-worker-w6-39fb96-openshift-default-5f55489475-z6jxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body=
Oct 30 21:02:28.594875 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:28.594825    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-39fb96-openshift-default-5f55489475-z6jxt" podUID="d3990d1c-906f-4267-b67b-5b1309f20cfc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused"
Oct 30 21:02:28.672530 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:28.672499    4521 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-9zp68"
Oct 30 21:02:28.694344 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:28.692375    4521 patch_prober.go:29] interesting pod/e2e-worker-w5-65968b-openshift-default-68896fcddc-9zp68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body=
Oct 30 21:02:28.694344 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:28.692443    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-9zp68" podUID="65e4dd7b-506f-4350-a82c-e35e61706595" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused"
Oct 30 21:02:28.716389 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:28.702827    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28e05ac8-5b4d-4085-bf8c-58793ca1c370-maas-api-xks-ca" pod "28e05ac8-5b4d-4085-bf8c-58793ca1c370" (UID: "28e05ac8-5b4d-4085-bf8c-58793ca1c370"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:02:28.801729 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:28.781849    4521 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28e05ac8-5b4d-4085-bf8c-58793ca1c370-maas-api-xks-ca\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:28.840740 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:28.840708    4521 patch_prober.go:29] interesting pod/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-nvwt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body=
Oct 30 21:02:28.857369 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:28.840988    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-nvwt8" podUID="6cff2b61-dcbd-4450-b776-43eff57a4fd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused"
Oct 30 21:02:28.996364 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:28.989029    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-sbpfk"]
Oct 30 21:02:29.016352 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:29.014785    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-65968b/premium-w5-65968b-kserve-67977485f7-c9jpv" podUID="92b2dd65-8a83-436d-afa8-50e5f0d11fa0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 30 21:02:29.027988 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:29.027952    4521 patch_prober.go:29] interesting pod/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-m9wn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Oct 30 21:02:29.028250 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:29.028222    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-m9wn4" podUID="9dc1356e-e12d-49f9-822e-f1ccf4fdaa3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Oct 30 21:02:29.059314 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:29.059218    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a0c5b/premium-w0-4a0c5b-kserve-85dcf89f57-mscdn" podUID="00e42758-77ce-4663-adfa-2202c0aae24a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:02:29.069118 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:29.069077    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a8f94/simulator-w3-5a8f94-kserve-77cc88cc89-n4sl6" podUID="1fa6900c-eee2-4f00-b0ec-248def48dfc4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:02:29.069679 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:29.069649    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a8f94/premium-w3-5a8f94-kserve-bb96b6848-5hk8h" podUID="24ccc37d-027a-4097-8da5-04df417aef94" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Oct 30 21:02:29.140466 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:29.138258    4521 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-9zp68" podStartSLOduration=3.138220991 podStartE2EDuration="3.138220991s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:02:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:02:28.434059247 +0000 UTC m=+1415.779680486" watchObservedRunningTime="2026-10-30 21:02:29.138220991 +0000 UTC m=+1416.483842230"
Oct 30 21:02:29.235585 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:29.235540    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-214c7679-bnjxj"
Oct 30 21:02:29.238108 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:29.238081    4521 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-c2d1eb/premium-w4-c2d1eb-kserve-8696fbd9b9-mxqhm"]
Oct 30 21:02:29.246246 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:29.246166    4521 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-39fb96-openshift-default-5f55489475-z6jxt" podStartSLOduration=3.24614798 podStartE2EDuration="3.24614798s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:02:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:02:28.568441325 +0000 UTC m=+1415.914062562" watchObservedRunningTime="2026-10-30 21:02:29.24614798 +0000 UTC m=+1416.591769220"
Oct 30 21:02:29.331659 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:29.330457    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bd27e7e-be97-4796-84dd-72f42b5bf42e-kube-api-access-hnszs\" (UniqueName: \"kubernetes.io/projected/7bd27e7e-be97-4796-84dd-72f42b5bf42e-kube-api-access-hnszs\") pod \"7bd27e7e-be97-4796-84dd-72f42b5bf42e\" (UID: \"7bd27e7e-be97-4796-84dd-72f42b5bf42e\") "
Oct 30 21:02:29.331659 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:29.330549    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7bd27e7e-be97-4796-84dd-72f42b5bf42e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bd27e7e-be97-4796-84dd-72f42b5bf42e-maas-api-openshift-ca\") pod \"7bd27e7e-be97-4796-84dd-72f42b5bf42e\" (UID: \"7bd27e7e-be97-4796-84dd-72f42b5bf42e\") "
Oct 30 21:02:29.331659 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:29.330618    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7bd27e7e-be97-4796-84dd-72f42b5bf42e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bd27e7e-be97-4796-84dd-72f42b5bf42e-maas-api-xks-ca\") pod \"7bd27e7e-be97-4796-84dd-72f42b5bf42e\" (UID: \"7bd27e7e-be97-4796-84dd-72f42b5bf42e\") "
Oct 30 21:02:29.352400 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:29.329917    4521 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-32a969-49xn7"]
Oct 30 21:02:29.354975 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:29.354940    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bd27e7e-be97-4796-84dd-72f42b5bf42e-maas-api-openshift-ca" pod "7bd27e7e-be97-4796-84dd-72f42b5bf42e" (UID: "7bd27e7e-be97-4796-84dd-72f42b5bf42e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:02:29.360827 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:29.360795    4521 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-32a969-49xn7"]
Oct 30 21:02:29.400188 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:29.389487    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bd27e7e-be97-4796-84dd-72f42b5bf42e-kube-api-access-hnszs" pod "7bd27e7e-be97-4796-84dd-72f42b5bf42e" (UID: "7bd27e7e-be97-4796-84dd-72f42b5bf42e"). InnerVolumeSpecName "kube-api-access-hnszs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:02:29.400188 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:29.389583    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7bd27e7e-be97-4796-84dd-72f42b5bf42e-maas-api-xks-ca" pod "7bd27e7e-be97-4796-84dd-72f42b5bf42e" (UID: "7bd27e7e-be97-4796-84dd-72f42b5bf42e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:02:29.631736 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:29.602918    4521 patch_prober.go:29] interesting pod/e2e-worker-w6-39fb96-openshift-default-5f55489475-z6jxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body=
Oct 30 21:02:29.631736 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:29.602978    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-39fb96-openshift-default-5f55489475-z6jxt" podUID="d3990d1c-906f-4267-b67b-5b1309f20cfc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused"
Oct 30 21:02:29.658423 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:29.658387    4521 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hnszs\" (UniqueName: \"kubernetes.io/projected/7bd27e7e-be97-4796-84dd-72f42b5bf42e-kube-api-access-hnszs\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:29.658650 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:29.658628    4521 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bd27e7e-be97-4796-84dd-72f42b5bf42e-maas-api-openshift-ca\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:29.658795 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:29.658780    4521 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bd27e7e-be97-4796-84dd-72f42b5bf42e-maas-api-xks-ca\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:29.714831 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:29.696775    4521 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-4a0c5b/premium-w0-4a0c5b-kserve-85dcf89f57-mscdn" podUID="00e42758-77ce-4663-adfa-2202c0aae24a" containerName="main" containerID="cri-o://ecc44c3983a6afc4815071bbc2e79af31a0e055659cee676c6271a9969018ae1" gracePeriod=45
Oct 30 21:02:29.714831 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:29.697092    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a8f94/distinct-2-w3-5a8f94-kserve-ffcc6449b-hv48k" podUID="bc0f5352-5c44-4c35-adf5-3fffab06b644" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:02:29.738414 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:29.735948    4521 patch_prober.go:29] interesting pod/e2e-worker-w5-65968b-openshift-default-68896fcddc-9zp68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body=
Oct 30 21:02:29.738414 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:29.736003    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-9zp68" podUID="65e4dd7b-506f-4350-a82c-e35e61706595" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused"
Oct 30 21:02:29.853117 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:29.852654    4521 patch_prober.go:29] interesting pod/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-nvwt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body=
Oct 30 21:02:29.853117 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:29.852720    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-nvwt8" podUID="6cff2b61-dcbd-4450-b776-43eff57a4fd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused"
Oct 30 21:02:29.910544 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:29.905775    4521 patch_prober.go:29] interesting pod/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-m9wn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Oct 30 21:02:29.910544 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:29.905825    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-m9wn4" podUID="9dc1356e-e12d-49f9-822e-f1ccf4fdaa3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Oct 30 21:02:29.910544 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:29.907169    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-214c7679-bnjxj" event={"ID":"7bd27e7e-be97-4796-84dd-72f42b5bf42e","Type":"ContainerDied","Data":"50f6973008f1bd07bdb44bb23364213d93e45c7571778b752cc7de33b003c49c"}
Oct 30 21:02:29.910544 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:29.907241    4521 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="50f6973008f1bd07bdb44bb23364213d93e45c7571778b752cc7de33b003c49c"
Oct 30 21:02:29.912523 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:29.912448    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-214c7679-bnjxj"
Oct 30 21:02:29.970369 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:29.969507    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-sbpfk" event={"ID":"7a5f91bb-7229-454f-a006-0d768f8879f4","Type":"ContainerStarted","Data":"805ef8c72c4f1574bdbfa9bbe04ce8260c0199472a8325633fd65adb6b7f588c"}
Oct 30 21:02:29.970369 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:29.969613    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-sbpfk" event={"ID":"7a5f91bb-7229-454f-a006-0d768f8879f4","Type":"ContainerStarted","Data":"03fcb95c00ed7b9c5b71c9219fac7330d82cfd09d91d63a03fce277ebf015d09"}
Oct 30 21:02:30.056124 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:30.046749    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a0c5b/premium-w0-4a0c5b-kserve-85dcf89f57-mscdn" podUID="00e42758-77ce-4663-adfa-2202c0aae24a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": EOF"
Oct 30 21:02:30.221979 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:30.213770    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a0c5b/premium-w0-4a0c5b-kserve-85dcf89f57-mscdn" podUID="00e42758-77ce-4663-adfa-2202c0aae24a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": dial tcp 10.133.0.52:8000: connect: connection refused"
Oct 30 21:02:30.221979 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:30.217599    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4a0c5b/premium-w0-4a0c5b-kserve-85dcf89f57-mscdn"
Oct 30 21:02:30.249779 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:30.238409    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6f6a826-ggh4m"
Oct 30 21:02:30.349134 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:30.346676    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a8f94/distinct-w3-5a8f94-kserve-5f8b544d59-mf95j" podUID="b8ac1cca-4ebd-474c-b32f-a3595a4cd847" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:02:30.626762 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:30.626637    4521 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-sbpfk" podStartSLOduration=4.626614489 podStartE2EDuration="4.626614489s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:02:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:02:30.406629421 +0000 UTC m=+1417.752250663" watchObservedRunningTime="2026-10-30 21:02:30.626614489 +0000 UTC m=+1417.972235728"
Oct 30 21:02:30.713568 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:30.692424    4521 patch_prober.go:29] interesting pod/e2e-worker-w5-65968b-openshift-default-68896fcddc-9zp68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body=
Oct 30 21:02:30.713568 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:30.692487    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-9zp68" podUID="65e4dd7b-506f-4350-a82c-e35e61706595" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused"
Oct 30 21:02:30.722706 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:30.721958    4521 patch_prober.go:29] interesting pod/e2e-worker-w6-39fb96-openshift-default-5f55489475-z6jxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body=
Oct 30 21:02:30.722706 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:30.722020    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-39fb96-openshift-default-5f55489475-z6jxt" podUID="d3990d1c-906f-4267-b67b-5b1309f20cfc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused"
Oct 30 21:02:30.755344 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:30.729833    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b9a4c83-9045-460b-bec6-8c2a6c3615a3-kube-api-access-sfwlq\" (UniqueName: \"kubernetes.io/projected/9b9a4c83-9045-460b-bec6-8c2a6c3615a3-kube-api-access-sfwlq\") pod \"9b9a4c83-9045-460b-bec6-8c2a6c3615a3\" (UID: \"9b9a4c83-9045-460b-bec6-8c2a6c3615a3\") "
Oct 30 21:02:30.755344 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:30.730015    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b9a4c83-9045-460b-bec6-8c2a6c3615a3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b9a4c83-9045-460b-bec6-8c2a6c3615a3-maas-api-xks-ca\") pod \"9b9a4c83-9045-460b-bec6-8c2a6c3615a3\" (UID: \"9b9a4c83-9045-460b-bec6-8c2a6c3615a3\") "
Oct 30 21:02:30.755344 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:30.730140    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b9a4c83-9045-460b-bec6-8c2a6c3615a3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b9a4c83-9045-460b-bec6-8c2a6c3615a3-maas-api-openshift-ca\") pod \"9b9a4c83-9045-460b-bec6-8c2a6c3615a3\" (UID: \"9b9a4c83-9045-460b-bec6-8c2a6c3615a3\") "
Oct 30 21:02:30.755344 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:30.742748    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b9a4c83-9045-460b-bec6-8c2a6c3615a3-maas-api-openshift-ca" pod "9b9a4c83-9045-460b-bec6-8c2a6c3615a3" (UID: "9b9a4c83-9045-460b-bec6-8c2a6c3615a3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:02:30.755344 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:30.743623    4521 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b9a4c83-9045-460b-bec6-8c2a6c3615a3-maas-api-openshift-ca\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:30.816603 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:30.781935    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b9a4c83-9045-460b-bec6-8c2a6c3615a3-maas-api-xks-ca" pod "9b9a4c83-9045-460b-bec6-8c2a6c3615a3" (UID: "9b9a4c83-9045-460b-bec6-8c2a6c3615a3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:02:30.830741 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:30.826495    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b9a4c83-9045-460b-bec6-8c2a6c3615a3-kube-api-access-sfwlq" pod "9b9a4c83-9045-460b-bec6-8c2a6c3615a3" (UID: "9b9a4c83-9045-460b-bec6-8c2a6c3615a3"). InnerVolumeSpecName "kube-api-access-sfwlq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:02:30.838356 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:30.831006    4521 patch_prober.go:29] interesting pod/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-nvwt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body=
Oct 30 21:02:30.838356 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:30.831072    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-nvwt8" podUID="6cff2b61-dcbd-4450-b776-43eff57a4fd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused"
Oct 30 21:02:30.846707 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:30.846633    4521 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfwlq\" (UniqueName: \"kubernetes.io/projected/9b9a4c83-9045-460b-bec6-8c2a6c3615a3-kube-api-access-sfwlq\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:30.846707 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:30.846675    4521 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b9a4c83-9045-460b-bec6-8c2a6c3615a3-maas-api-xks-ca\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:30.935422 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:30.922513    4521 patch_prober.go:29] interesting pod/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-m9wn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Oct 30 21:02:30.935422 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:30.922576    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-m9wn4" podUID="9dc1356e-e12d-49f9-822e-f1ccf4fdaa3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Oct 30 21:02:30.937922 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:30.937779    4521 patch_prober.go:29] interesting pod/e2e-worker-w5-65968b-openshift-default-68896fcddc-sbpfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body=
Oct 30 21:02:30.937922 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:30.937839    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-sbpfk" podUID="7a5f91bb-7229-454f-a006-0d768f8879f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused"
Oct 30 21:02:31.173387 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:31.171507    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a0c5b/premium-w0-4a0c5b-kserve-85dcf89f57-mscdn" podUID="00e42758-77ce-4663-adfa-2202c0aae24a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": dial tcp 10.133.0.52:8000: connect: connection refused"
Oct 30 21:02:31.237407 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:31.177414    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a8f94/premium-w3-5a8f94-kserve-bb96b6848-5hk8h" podUID="24ccc37d-027a-4097-8da5-04df417aef94" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Oct 30 21:02:31.237407 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:31.182410    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-65968b/premium-w5-65968b-kserve-67977485f7-c9jpv" podUID="92b2dd65-8a83-436d-afa8-50e5f0d11fa0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 30 21:02:31.237407 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:31.182696    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a8f94/simulator-w3-5a8f94-kserve-77cc88cc89-n4sl6" podUID="1fa6900c-eee2-4f00-b0ec-248def48dfc4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 30 21:02:31.237407 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:31.233157    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-65968b/distinct-2-w5-65968b-kserve-6d65c85f6c-dz5r7" podUID="d6751cf5-7df8-4ffa-b6c8-8b0c4f16ba2a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": context deadline exceeded"
Oct 30 21:02:31.244798 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:31.241625    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c4b379/premium-w1-c4b379-kserve-575f896d49-2n6gz" podUID="2aac3e8e-d631-4b9c-bdbb-3b4b1df133a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:02:31.347458 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:31.301943    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6f6a826-ggh4m"
Oct 30 21:02:31.373857 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:31.373814    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a8f94/distinct-w3-5a8f94-kserve-5f8b544d59-mf95j" podUID="b8ac1cca-4ebd-474c-b32f-a3595a4cd847" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:02:31.402352 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:31.302615    4521 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="080ab3d3-8e29-4214-9b65-b9a2ce6f0f3a" path="/var/lib/kubelet/pods/080ab3d3-8e29-4214-9b65-b9a2ce6f0f3a/volumes"
Oct 30 21:02:31.402352 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:31.389738    4521 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-worker-w6-39fb96-openshift-default-5f55489475-z6jxt" secret="" err="secret \"e2e-worker-w6-39fb96-openshift-default-dockercfg-t7kpm\" not found"
Oct 30 21:02:31.438448 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:31.438406    4521 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-sbpfk"
Oct 30 21:02:31.458444 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:31.458405    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6f6a826-ggh4m" event={"ID":"9b9a4c83-9045-460b-bec6-8c2a6c3615a3","Type":"ContainerDied","Data":"50be316b11757f6a4f705635ca3f792dfc292b49e9834c2231b0862ed096c811"}
Oct 30 21:02:31.458710 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:31.458692    4521 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="50be316b11757f6a4f705635ca3f792dfc292b49e9834c2231b0862ed096c811"
Oct 30 21:02:31.472348 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:31.462407    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5a8f94/distinct-w3-5a8f94-kserve-5f8b544d59-mf95j"
Oct 30 21:02:31.622702 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:31.602603    4521 patch_prober.go:29] interesting pod/e2e-worker-w6-39fb96-openshift-default-5f55489475-z6jxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body=
Oct 30 21:02:31.622702 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:31.602660    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-39fb96-openshift-default-5f55489475-z6jxt" podUID="d3990d1c-906f-4267-b67b-5b1309f20cfc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused"
Oct 30 21:02:31.622702 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:31.620516    4521 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-39fb96-openshift-default-5f55489475-z6jxt"]
Oct 30 21:02:31.661569 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:31.661535    4521 patch_prober.go:29] interesting pod/e2e-worker-w5-65968b-openshift-default-68896fcddc-9zp68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body=
Oct 30 21:02:31.661801 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:31.661776    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-9zp68" podUID="65e4dd7b-506f-4350-a82c-e35e61706595" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused"
Oct 30 21:02:31.671461 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:31.671415    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a8f94/distinct-2-w3-5a8f94-kserve-ffcc6449b-hv48k" podUID="bc0f5352-5c44-4c35-adf5-3fffab06b644" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:02:31.703107 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:31.703060    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c4b379/distinct-2-w1-c4b379-kserve-7999b785f5-97fz5" podUID="080a654d-a5bf-4845-a07f-cc8a7e6d0ace" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:02:31.860356 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:31.835169    4521 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-39fb96-openshift-default-5f55489475-ctbdp"]
Oct 30 21:02:31.860900 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:31.860858    4521 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-39fb96-openshift-default-5f55489475-ctbdp" podUID="55990440-6bb2-4ce7-945f-6ada2906a855" containerName="istio-proxy" containerID="cri-o://f3f770366013af52bebfc6b7500a37d53693245530cfa7a6c52c4aa6ebbbbee0" gracePeriod=30
Oct 30 21:02:31.861203 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:31.844459    4521 patch_prober.go:29] interesting pod/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-nvwt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body=
Oct 30 21:02:31.866102 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:31.866054    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-nvwt8" podUID="6cff2b61-dcbd-4450-b776-43eff57a4fd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused"
Oct 30 21:02:31.895355 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:31.895301    4521 patch_prober.go:29] interesting pod/e2e-worker-w5-65968b-openshift-default-68896fcddc-sbpfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body=
Oct 30 21:02:31.895676 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:31.895628    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-sbpfk" podUID="7a5f91bb-7229-454f-a006-0d768f8879f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused"
Oct 30 21:02:31.916636 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:31.915060    4521 patch_prober.go:29] interesting pod/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-m9wn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Oct 30 21:02:31.916636 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:31.915124    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-m9wn4" podUID="9dc1356e-e12d-49f9-822e-f1ccf4fdaa3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Oct 30 21:02:32.033820 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:32.008770    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a0c5b/premium-w0-4a0c5b-kserve-85dcf89f57-mscdn" podUID="00e42758-77ce-4663-adfa-2202c0aae24a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": dial tcp 10.133.0.52:8000: connect: connection refused"
Oct 30 21:02:32.204413 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:32.180416    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a8f94/premium-w3-5a8f94-kserve-bb96b6848-5hk8h" podUID="24ccc37d-027a-4097-8da5-04df417aef94" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:02:32.204413 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:32.180653    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5a8f94/premium-w3-5a8f94-kserve-bb96b6848-5hk8h"
Oct 30 21:02:32.204413 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:32.201076    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a8f94/simulator-w3-5a8f94-kserve-77cc88cc89-n4sl6" podUID="1fa6900c-eee2-4f00-b0ec-248def48dfc4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:02:32.204413 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:32.201446    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-65968b/premium-w5-65968b-kserve-67977485f7-c9jpv" podUID="92b2dd65-8a83-436d-afa8-50e5f0d11fa0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 30 21:02:32.204413 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:32.201767    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5a8f94/simulator-w3-5a8f94-kserve-77cc88cc89-n4sl6"
Oct 30 21:02:32.204413 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:32.201827    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-65968b/premium-w5-65968b-kserve-67977485f7-c9jpv"
Oct 30 21:02:32.251346 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:32.250632    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c4b379/premium-w1-c4b379-kserve-575f896d49-2n6gz" podUID="2aac3e8e-d631-4b9c-bdbb-3b4b1df133a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 30 21:02:32.251346 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:32.251080    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c4b379/premium-w1-c4b379-kserve-575f896d49-2n6gz"
Oct 30 21:02:32.307353 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:32.288397    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-65968b/distinct-2-w5-65968b-kserve-6d65c85f6c-dz5r7" podUID="d6751cf5-7df8-4ffa-b6c8-8b0c4f16ba2a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:02:32.307353 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:32.288788    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-65968b/distinct-2-w5-65968b-kserve-6d65c85f6c-dz5r7"
Oct 30 21:02:32.378301 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:32.309128    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c2d1eb/premium-w4-c2d1eb-kserve-8696fbd9b9-mxqhm" podUID="4bb0f898-b77f-419d-bff4-6413e82ef51d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:02:32.403796 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:32.386426    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a8f94/distinct-w3-5a8f94-kserve-5f8b544d59-mf95j" podUID="b8ac1cca-4ebd-474c-b32f-a3595a4cd847" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:02:32.403796 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:32.396629    4521 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-c4b379/premium-w1-c4b379-kserve-575f896d49-2n6gz" podUID="2aac3e8e-d631-4b9c-bdbb-3b4b1df133a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:02:32.472406 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:32.454403    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4kxb"]
Oct 30 21:02:32.587298 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:32.586843    4521 patch_prober.go:29] interesting pod/e2e-worker-w6-39fb96-openshift-default-5f55489475-z6jxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body=
Oct 30 21:02:32.587298 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:32.586906    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-39fb96-openshift-default-5f55489475-z6jxt" podUID="d3990d1c-906f-4267-b67b-5b1309f20cfc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused"
Oct 30 21:02:32.660360 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:32.660218    4521 patch_prober.go:29] interesting pod/e2e-worker-w5-65968b-openshift-default-68896fcddc-9zp68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body=
Oct 30 21:02:32.660360 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:32.660282    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-9zp68" podUID="65e4dd7b-506f-4350-a82c-e35e61706595" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused"
Oct 30 21:02:32.728454 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:32.701439    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a8f94/distinct-2-w3-5a8f94-kserve-ffcc6449b-hv48k" podUID="bc0f5352-5c44-4c35-adf5-3fffab06b644" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:02:32.728454 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:32.702254    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5a8f94/distinct-2-w3-5a8f94-kserve-ffcc6449b-hv48k"
Oct 30 21:02:32.728454 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:32.725605    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c4b379/distinct-2-w1-c4b379-kserve-7999b785f5-97fz5" podUID="080a654d-a5bf-4845-a07f-cc8a7e6d0ace" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:02:32.728454 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:32.726035    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c4b379/distinct-2-w1-c4b379-kserve-7999b785f5-97fz5"
Oct 30 21:02:32.878365 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:32.855623    4521 patch_prober.go:29] interesting pod/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-nvwt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body=
Oct 30 21:02:32.878365 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:32.855713    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-nvwt8" podUID="6cff2b61-dcbd-4450-b776-43eff57a4fd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused"
Oct 30 21:02:33.024541 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:32.941842    4521 patch_prober.go:29] interesting pod/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-m9wn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Oct 30 21:02:33.024541 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:32.941922    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-m9wn4" podUID="9dc1356e-e12d-49f9-822e-f1ccf4fdaa3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Oct 30 21:02:33.024541 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:32.942014    4521 patch_prober.go:29] interesting pod/e2e-worker-w5-65968b-openshift-default-68896fcddc-sbpfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body=
Oct 30 21:02:33.024541 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:32.942040    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-sbpfk" podUID="7a5f91bb-7229-454f-a006-0d768f8879f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused"
Oct 30 21:02:33.024541 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:33.011748    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a0c5b/premium-w0-4a0c5b-kserve-85dcf89f57-mscdn" podUID="00e42758-77ce-4663-adfa-2202c0aae24a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": dial tcp 10.133.0.52:8000: connect: connection refused"
Oct 30 21:02:33.253350 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:33.245590    4521 generic.go:356] "Generic (PLEG): container finished" podID="00e42758-77ce-4663-adfa-2202c0aae24a" containerID="ecc44c3983a6afc4815071bbc2e79af31a0e055659cee676c6271a9969018ae1" exitCode=0
Oct 30 21:02:33.257475 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:33.257424    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c4b379/premium-w1-c4b379-kserve-575f896d49-2n6gz" podUID="2aac3e8e-d631-4b9c-bdbb-3b4b1df133a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 30 21:02:33.352602 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:33.345359    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c2d1eb/premium-w4-c2d1eb-kserve-8696fbd9b9-mxqhm" podUID="4bb0f898-b77f-419d-bff4-6413e82ef51d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded"
Oct 30 21:02:33.411348 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:33.363443    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a8f94/premium-w3-5a8f94-kserve-bb96b6848-5hk8h" podUID="24ccc37d-027a-4097-8da5-04df417aef94" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:02:33.602113 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:33.381627    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-65968b/distinct-2-w5-65968b-kserve-6d65c85f6c-dz5r7" podUID="d6751cf5-7df8-4ffa-b6c8-8b0c4f16ba2a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": context deadline exceeded"
Oct 30 21:02:33.602113 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:33.408621    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-65968b/premium-w5-65968b-kserve-67977485f7-c9jpv" podUID="92b2dd65-8a83-436d-afa8-50e5f0d11fa0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 30 21:02:33.602113 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:33.408962    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a8f94/simulator-w3-5a8f94-kserve-77cc88cc89-n4sl6" podUID="1fa6900c-eee2-4f00-b0ec-248def48dfc4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:02:33.615676 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:33.452887    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a8f94/distinct-w3-5a8f94-kserve-5f8b544d59-mf95j" podUID="b8ac1cca-4ebd-474c-b32f-a3595a4cd847" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:02:33.615676 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:33.455190    4521 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-39fb96-openshift-default-5f55489475-z6jxt" podUID="d3990d1c-906f-4267-b67b-5b1309f20cfc" containerName="istio-proxy" containerID="cri-o://d2435b724e4f5fc6ce200c74e76d64b9c4188f66739b19ce5da0d0800f4797fa" gracePeriod=30
Oct 30 21:02:33.648052 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:33.648006    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c2d1eb/premium-w4-c2d1eb-kserve-8696fbd9b9-mxqhm"
Oct 30 21:02:33.776616 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:33.776578    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-4a0c5b/premium-w0-4a0c5b-kserve-85dcf89f57-mscdn" event={"ID":"00e42758-77ce-4663-adfa-2202c0aae24a","Type":"ContainerDied","Data":"ecc44c3983a6afc4815071bbc2e79af31a0e055659cee676c6271a9969018ae1"}
Oct 30 21:02:33.776880 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:33.703472    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a8f94/distinct-2-w3-5a8f94-kserve-ffcc6449b-hv48k" podUID="bc0f5352-5c44-4c35-adf5-3fffab06b644" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:02:33.785593 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:33.731529    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c4b379/distinct-2-w1-c4b379-kserve-7999b785f5-97fz5" podUID="080a654d-a5bf-4845-a07f-cc8a7e6d0ace" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:02:33.799854 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:33.663592    4521 patch_prober.go:29] interesting pod/e2e-worker-w5-65968b-openshift-default-68896fcddc-9zp68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body=
Oct 30 21:02:33.800157 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:33.800110    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-9zp68" podUID="65e4dd7b-506f-4350-a82c-e35e61706595" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused"
Oct 30 21:02:33.885114 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:33.885015    4521 patch_prober.go:29] interesting pod/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-nvwt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body=
Oct 30 21:02:33.901532 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:33.893833    4521 patch_prober.go:29] interesting pod/e2e-worker-w5-65968b-openshift-default-68896fcddc-sbpfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body=
Oct 30 21:02:33.901836 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:33.901784    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-sbpfk" podUID="7a5f91bb-7229-454f-a006-0d768f8879f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused"
Oct 30 21:02:33.902051 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:33.902017    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-nvwt8" podUID="6cff2b61-dcbd-4450-b776-43eff57a4fd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused"
Oct 30 21:02:33.913649 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:33.913608    4521 patch_prober.go:29] interesting pod/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-m9wn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Oct 30 21:02:33.919471 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:33.919427    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-m9wn4" podUID="9dc1356e-e12d-49f9-822e-f1ccf4fdaa3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Oct 30 21:02:34.259505 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:34.259452    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c4b379/premium-w1-c4b379-kserve-575f896d49-2n6gz" podUID="2aac3e8e-d631-4b9c-bdbb-3b4b1df133a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:02:34.464349 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:34.439419    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-4a0c5b/premium-w0-4a0c5b-kserve-85dcf89f57-mscdn" event={"ID":"00e42758-77ce-4663-adfa-2202c0aae24a","Type":"ContainerDied","Data":"bec9b5108a5763f427eafb09de63f6ae882559cbc1f2006e8e6fa73ec57f479f"}
Oct 30 21:02:34.522352 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:34.509468    4521 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bec9b5108a5763f427eafb09de63f6ae882559cbc1f2006e8e6fa73ec57f479f"
Oct 30 21:02:34.522352 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:34.466535    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c2d1eb/premium-w4-c2d1eb-kserve-8696fbd9b9-mxqhm" podUID="4bb0f898-b77f-419d-bff4-6413e82ef51d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded"
Oct 30 21:02:34.522352 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:34.439664    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-4a0c5b/premium-w0-4a0c5b-kserve-85dcf89f57-mscdn"
Oct 30 21:02:34.615923 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:34.615555    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a8f94/distinct-w3-5a8f94-kserve-5f8b544d59-mf95j" podUID="b8ac1cca-4ebd-474c-b32f-a3595a4cd847" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:02:34.616583 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:34.616544    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a8f94/simulator-w3-5a8f94-kserve-77cc88cc89-n4sl6" podUID="1fa6900c-eee2-4f00-b0ec-248def48dfc4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 30 21:02:34.632398 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:34.621600    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-65968b/distinct-2-w5-65968b-kserve-6d65c85f6c-dz5r7" podUID="d6751cf5-7df8-4ffa-b6c8-8b0c4f16ba2a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:02:34.633729 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:34.633681    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a8f94/premium-w3-5a8f94-kserve-bb96b6848-5hk8h" podUID="24ccc37d-027a-4097-8da5-04df417aef94" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Oct 30 21:02:34.639593 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:34.635127    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-65968b/premium-w5-65968b-kserve-67977485f7-c9jpv" podUID="92b2dd65-8a83-436d-afa8-50e5f0d11fa0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 30 21:02:34.639593 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:34.637853    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00e42758-77ce-4663-adfa-2202c0aae24a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/00e42758-77ce-4663-adfa-2202c0aae24a-dshm\") pod \"00e42758-77ce-4663-adfa-2202c0aae24a\" (UID: \"00e42758-77ce-4663-adfa-2202c0aae24a\") "
Oct 30 21:02:34.639593 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:34.638025    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00e42758-77ce-4663-adfa-2202c0aae24a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00e42758-77ce-4663-adfa-2202c0aae24a-tmp-dir\") pod \"00e42758-77ce-4663-adfa-2202c0aae24a\" (UID: \"00e42758-77ce-4663-adfa-2202c0aae24a\") "
Oct 30 21:02:34.639593 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:34.638125    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00e42758-77ce-4663-adfa-2202c0aae24a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00e42758-77ce-4663-adfa-2202c0aae24a-model-cache\") pod \"00e42758-77ce-4663-adfa-2202c0aae24a\" (UID: \"00e42758-77ce-4663-adfa-2202c0aae24a\") "
Oct 30 21:02:34.639593 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:34.638210    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00e42758-77ce-4663-adfa-2202c0aae24a-kube-api-access-z942d\" (UniqueName: \"kubernetes.io/projected/00e42758-77ce-4663-adfa-2202c0aae24a-kube-api-access-z942d\") pod \"00e42758-77ce-4663-adfa-2202c0aae24a\" (UID: \"00e42758-77ce-4663-adfa-2202c0aae24a\") "
Oct 30 21:02:34.639593 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:34.638244    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/00e42758-77ce-4663-adfa-2202c0aae24a-tls-certs\" (UniqueName: \"kubernetes.io/secret/00e42758-77ce-4663-adfa-2202c0aae24a-tls-certs\") pod \"00e42758-77ce-4663-adfa-2202c0aae24a\" (UID: \"00e42758-77ce-4663-adfa-2202c0aae24a\") "
Oct 30 21:02:34.639593 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:34.638278    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00e42758-77ce-4663-adfa-2202c0aae24a-home\" (UniqueName: \"kubernetes.io/empty-dir/00e42758-77ce-4663-adfa-2202c0aae24a-home\") pod \"00e42758-77ce-4663-adfa-2202c0aae24a\" (UID: \"00e42758-77ce-4663-adfa-2202c0aae24a\") "
Oct 30 21:02:34.639593 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:34.639248    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00e42758-77ce-4663-adfa-2202c0aae24a-home" pod "00e42758-77ce-4663-adfa-2202c0aae24a" (UID: "00e42758-77ce-4663-adfa-2202c0aae24a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:02:34.669967 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:34.669922    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00e42758-77ce-4663-adfa-2202c0aae24a-tmp-dir" pod "00e42758-77ce-4663-adfa-2202c0aae24a" (UID: "00e42758-77ce-4663-adfa-2202c0aae24a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:02:34.670253 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:34.670230    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00e42758-77ce-4663-adfa-2202c0aae24a-model-cache" pod "00e42758-77ce-4663-adfa-2202c0aae24a" (UID: "00e42758-77ce-4663-adfa-2202c0aae24a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:02:34.693427 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:34.693392    4521 patch_prober.go:29] interesting pod/e2e-worker-w5-65968b-openshift-default-68896fcddc-9zp68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body=
Oct 30 21:02:34.693695 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:34.693667    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-9zp68" podUID="65e4dd7b-506f-4350-a82c-e35e61706595" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused"
Oct 30 21:02:34.716709 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:34.716622    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00e42758-77ce-4663-adfa-2202c0aae24a-dshm" pod "00e42758-77ce-4663-adfa-2202c0aae24a" (UID: "00e42758-77ce-4663-adfa-2202c0aae24a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:02:34.758859 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:34.758827    4521 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00e42758-77ce-4663-adfa-2202c0aae24a-tmp-dir\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:34.759601 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:34.759578    4521 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00e42758-77ce-4663-adfa-2202c0aae24a-model-cache\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:34.759755 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:34.759739    4521 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00e42758-77ce-4663-adfa-2202c0aae24a-home\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:34.759888 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:34.759876    4521 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00e42758-77ce-4663-adfa-2202c0aae24a-dshm\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:34.787349 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:34.783414    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a8f94/distinct-2-w3-5a8f94-kserve-ffcc6449b-hv48k" podUID="bc0f5352-5c44-4c35-adf5-3fffab06b644" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:02:34.953769 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:34.914553    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00e42758-77ce-4663-adfa-2202c0aae24a-kube-api-access-z942d" pod "00e42758-77ce-4663-adfa-2202c0aae24a" (UID: "00e42758-77ce-4663-adfa-2202c0aae24a"). InnerVolumeSpecName "kube-api-access-z942d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:02:34.988418 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:34.918834    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00e42758-77ce-4663-adfa-2202c0aae24a-tls-certs" pod "00e42758-77ce-4663-adfa-2202c0aae24a" (UID: "00e42758-77ce-4663-adfa-2202c0aae24a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:02:34.988418 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:34.920611    4521 patch_prober.go:29] interesting pod/e2e-worker-w5-65968b-openshift-default-68896fcddc-sbpfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body=
Oct 30 21:02:34.988418 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:34.987467    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-sbpfk" podUID="7a5f91bb-7229-454f-a006-0d768f8879f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused"
Oct 30 21:02:34.988418 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:34.920697    4521 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-c4b379/distinct-2-w1-c4b379-kserve-7999b785f5-97fz5" podUID="080a654d-a5bf-4845-a07f-cc8a7e6d0ace" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:02:34.988418 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:34.976025    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c4b379/distinct-2-w1-c4b379-kserve-7999b785f5-97fz5" podUID="080a654d-a5bf-4845-a07f-cc8a7e6d0ace" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": context deadline exceeded"
Oct 30 21:02:34.988418 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:34.920657    4521 patch_prober.go:29] interesting pod/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-nvwt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body=
Oct 30 21:02:34.988418 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:34.988209    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-nvwt8" podUID="6cff2b61-dcbd-4450-b776-43eff57a4fd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused"
Oct 30 21:02:34.988418 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:34.976365    4521 patch_prober.go:29] interesting pod/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-m9wn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Oct 30 21:02:34.988418 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:34.988261    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-m9wn4" podUID="9dc1356e-e12d-49f9-822e-f1ccf4fdaa3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Oct 30 21:02:34.988418 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:34.988349    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a0c5b/premium-w0-4a0c5b-kserve-85dcf89f57-mscdn" podUID="00e42758-77ce-4663-adfa-2202c0aae24a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:02:35.036611 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:35.027207    4521 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z942d\" (UniqueName: \"kubernetes.io/projected/00e42758-77ce-4663-adfa-2202c0aae24a-kube-api-access-z942d\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:35.036611 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:35.027248    4521 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00e42758-77ce-4663-adfa-2202c0aae24a-tls-certs\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:35.130944 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:35.130591    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5a8f94/distinct-2-w3-5a8f94-kserve-ffcc6449b-hv48k"
Oct 30 21:02:35.266117 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:35.266026    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c2d1eb/premium-w4-c2d1eb-kserve-8696fbd9b9-mxqhm"
Oct 30 21:02:35.289474 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:35.289410    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c4b379/premium-w1-c4b379-kserve-575f896d49-2n6gz" podUID="2aac3e8e-d631-4b9c-bdbb-3b4b1df133a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:02:35.612439 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:35.592989    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-4a0c5b/premium-w0-4a0c5b-kserve-85dcf89f57-mscdn"
Oct 30 21:02:35.648348 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:35.646672    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a8f94/distinct-w3-5a8f94-kserve-5f8b544d59-mf95j" podUID="b8ac1cca-4ebd-474c-b32f-a3595a4cd847" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded"
Oct 30 21:02:35.648853 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:35.648827    4521 patch_prober.go:29] interesting pod/e2e-worker-w5-65968b-openshift-default-68896fcddc-9zp68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body=
Oct 30 21:02:35.649165 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:35.649143    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-9zp68" podUID="65e4dd7b-506f-4350-a82c-e35e61706595" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused"
Oct 30 21:02:35.649302 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:35.649049    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a8f94/simulator-w3-5a8f94-kserve-77cc88cc89-n4sl6" podUID="1fa6900c-eee2-4f00-b0ec-248def48dfc4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:02:35.649679 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:35.649018    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-65968b/premium-w5-65968b-kserve-67977485f7-c9jpv" podUID="92b2dd65-8a83-436d-afa8-50e5f0d11fa0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:02:35.650801 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:35.649092    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a8f94/premium-w3-5a8f94-kserve-bb96b6848-5hk8h" podUID="24ccc37d-027a-4097-8da5-04df417aef94" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Oct 30 21:02:35.651183 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:35.649071    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-65968b/distinct-2-w5-65968b-kserve-6d65c85f6c-dz5r7" podUID="d6751cf5-7df8-4ffa-b6c8-8b0c4f16ba2a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": context deadline exceeded"
Oct 30 21:02:35.670968 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:35.670115    4521 patch_prober.go:29] interesting pod/e2e-worker-w6-39fb96-openshift-default-5f55489475-ctbdp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 21:02:35.670968 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:35.670873    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-39fb96-openshift-default-5f55489475-ctbdp" podUID="55990440-6bb2-4ce7-945f-6ada2906a855" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 21:02:35.829809 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:35.829768    4521 patch_prober.go:29] interesting pod/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-nvwt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body=
Oct 30 21:02:35.830063 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:35.830033    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-nvwt8" podUID="6cff2b61-dcbd-4450-b776-43eff57a4fd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused"
Oct 30 21:02:35.856349 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:35.785395    4521 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-4a0c5b/premium-w0-4a0c5b-kserve-85dcf89f57-mscdn"]
Oct 30 21:02:35.871462 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:35.871367    4521 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-4a0c5b/premium-w0-4a0c5b-kserve-85dcf89f57-mscdn"]
Oct 30 21:02:35.891379 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:35.882742    4521 patch_prober.go:29] interesting pod/e2e-worker-w5-65968b-openshift-default-68896fcddc-sbpfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body=
Oct 30 21:02:35.891379 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:35.882808    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-sbpfk" podUID="7a5f91bb-7229-454f-a006-0d768f8879f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused"
Oct 30 21:02:35.917349 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:35.907946    4521 patch_prober.go:29] interesting pod/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-m9wn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Oct 30 21:02:35.917349 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:35.908008    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-m9wn4" podUID="9dc1356e-e12d-49f9-822e-f1ccf4fdaa3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Oct 30 21:02:35.991476 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:35.991420    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c4b379/distinct-2-w1-c4b379-kserve-7999b785f5-97fz5" podUID="080a654d-a5bf-4845-a07f-cc8a7e6d0ace" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:02:36.280349 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:36.269641    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c4b379/distinct-2-w1-c4b379-kserve-7999b785f5-97fz5"
Oct 30 21:02:36.325460 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:36.325411    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c4b379/premium-w1-c4b379-kserve-575f896d49-2n6gz" podUID="2aac3e8e-d631-4b9c-bdbb-3b4b1df133a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 30 21:02:36.391503 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:36.325724    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c2d1eb/premium-w4-c2d1eb-kserve-8696fbd9b9-mxqhm" podUID="4bb0f898-b77f-419d-bff4-6413e82ef51d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:02:36.672531 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:36.672428    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-65968b/distinct-2-w5-65968b-kserve-6d65c85f6c-dz5r7" podUID="d6751cf5-7df8-4ffa-b6c8-8b0c4f16ba2a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": context deadline exceeded"
Oct 30 21:02:36.673059 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:36.673034    4521 patch_prober.go:29] interesting pod/e2e-worker-w5-65968b-openshift-default-68896fcddc-9zp68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body=
Oct 30 21:02:36.673229 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:36.673206    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-9zp68" podUID="65e4dd7b-506f-4350-a82c-e35e61706595" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused"
Oct 30 21:02:36.673375 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:36.672633    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a8f94/distinct-w3-5a8f94-kserve-5f8b544d59-mf95j" podUID="b8ac1cca-4ebd-474c-b32f-a3595a4cd847" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:02:36.677411 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:36.672660    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a8f94/simulator-w3-5a8f94-kserve-77cc88cc89-n4sl6" podUID="1fa6900c-eee2-4f00-b0ec-248def48dfc4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 30 21:02:36.677689 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:36.672600    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-65968b/premium-w5-65968b-kserve-67977485f7-c9jpv" podUID="92b2dd65-8a83-436d-afa8-50e5f0d11fa0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 30 21:02:36.678052 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:36.672692    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a8f94/premium-w3-5a8f94-kserve-bb96b6848-5hk8h" podUID="24ccc37d-027a-4097-8da5-04df417aef94" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Oct 30 21:02:36.722837 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:36.719434    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a8f94/distinct-2-w3-5a8f94-kserve-ffcc6449b-hv48k" podUID="bc0f5352-5c44-4c35-adf5-3fffab06b644" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:02:36.867351 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:36.856213    4521 patch_prober.go:29] interesting pod/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-nvwt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body=
Oct 30 21:02:36.867351 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:36.861419    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-nvwt8" podUID="6cff2b61-dcbd-4450-b776-43eff57a4fd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused"
Oct 30 21:02:36.899353 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:36.899191    4521 patch_prober.go:29] interesting pod/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-m9wn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Oct 30 21:02:36.899353 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:36.899254    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-m9wn4" podUID="9dc1356e-e12d-49f9-822e-f1ccf4fdaa3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Oct 30 21:02:36.904379 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:36.902373    4521 patch_prober.go:29] interesting pod/e2e-worker-w5-65968b-openshift-default-68896fcddc-sbpfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body=
Oct 30 21:02:36.904379 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:36.902423    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-sbpfk" podUID="7a5f91bb-7229-454f-a006-0d768f8879f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused"
Oct 30 21:02:36.932776 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:36.922341    4521 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00e42758-77ce-4663-adfa-2202c0aae24a" path="/var/lib/kubelet/pods/00e42758-77ce-4663-adfa-2202c0aae24a/volumes"
Oct 30 21:02:37.393569 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:37.393525    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c4b379/premium-w1-c4b379-kserve-575f896d49-2n6gz" podUID="2aac3e8e-d631-4b9c-bdbb-3b4b1df133a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:02:37.685487 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:37.683669    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-65968b/premium-w5-65968b-kserve-67977485f7-c9jpv"
Oct 30 21:02:37.685487 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:37.684253    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a8f94/distinct-w3-5a8f94-kserve-5f8b544d59-mf95j" podUID="b8ac1cca-4ebd-474c-b32f-a3595a4cd847" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:02:37.686033 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:37.685817    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-65968b/distinct-2-w5-65968b-kserve-6d65c85f6c-dz5r7" podUID="d6751cf5-7df8-4ffa-b6c8-8b0c4f16ba2a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": context deadline exceeded"
Oct 30 21:02:37.713460 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:37.686631    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a8f94/premium-w3-5a8f94-kserve-bb96b6848-5hk8h" podUID="24ccc37d-027a-4097-8da5-04df417aef94" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:02:37.713460 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:37.687099    4521 patch_prober.go:29] interesting pod/e2e-worker-w5-65968b-openshift-default-68896fcddc-9zp68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body=
Oct 30 21:02:37.713460 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:37.687136    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-9zp68" podUID="65e4dd7b-506f-4350-a82c-e35e61706595" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused"
Oct 30 21:02:37.829648 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:37.829595    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a8f94/simulator-w3-5a8f94-kserve-77cc88cc89-n4sl6" podUID="1fa6900c-eee2-4f00-b0ec-248def48dfc4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 30 21:02:37.854065 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:37.854012    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a8f94/distinct-2-w3-5a8f94-kserve-ffcc6449b-hv48k" podUID="bc0f5352-5c44-4c35-adf5-3fffab06b644" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": context deadline exceeded"
Oct 30 21:02:37.856184 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:37.856157    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5a8f94/distinct-2-w3-5a8f94-kserve-ffcc6449b-hv48k"
Oct 30 21:02:37.884068 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:37.884031    4521 patch_prober.go:29] interesting pod/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-nvwt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body=
Oct 30 21:02:38.045460 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:38.036534    4521 patch_prober.go:29] interesting pod/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-m9wn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Oct 30 21:02:38.045460 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:38.036602    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-m9wn4" podUID="9dc1356e-e12d-49f9-822e-f1ccf4fdaa3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Oct 30 21:02:38.056431 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:37.884312    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-nvwt8" podUID="6cff2b61-dcbd-4450-b776-43eff57a4fd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused"
Oct 30 21:02:38.251378 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:38.247206    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c2d1eb/premium-w4-c2d1eb-kserve-8696fbd9b9-mxqhm" podUID="4bb0f898-b77f-419d-bff4-6413e82ef51d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:02:38.277974 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:38.277938    4521 patch_prober.go:29] interesting pod/e2e-worker-w5-65968b-openshift-default-68896fcddc-sbpfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body=
Oct 30 21:02:38.278250 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:38.278218    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-sbpfk" podUID="7a5f91bb-7229-454f-a006-0d768f8879f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused"
Oct 30 21:02:38.524623 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:38.524582    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c4b379/premium-w1-c4b379-kserve-575f896d49-2n6gz" podUID="2aac3e8e-d631-4b9c-bdbb-3b4b1df133a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:02:38.756349 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:38.739144    4521 patch_prober.go:29] interesting pod/e2e-worker-w2-32a969-openshift-default-8945d87fc-hmwtj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:02:38.764359 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:38.759475    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-hmwtj" podUID="f6f47707-ed2c-4860-a13a-b1a5d9d1bbe3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:02:38.764359 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:38.760276    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5a8f94/simulator-w3-5a8f94-kserve-77cc88cc89-n4sl6"
Oct 30 21:02:38.852977 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:38.837720    4521 patch_prober.go:29] interesting pod/e2e-worker-w5-65968b-openshift-default-68896fcddc-9zp68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body=
Oct 30 21:02:38.852977 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:38.837779    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-9zp68" podUID="65e4dd7b-506f-4350-a82c-e35e61706595" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused"
Oct 30 21:02:38.852977 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:38.838467    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-65968b/premium-w5-65968b-kserve-67977485f7-c9jpv" podUID="92b2dd65-8a83-436d-afa8-50e5f0d11fa0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 30 21:02:38.852977 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:38.839493    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a8f94/premium-w3-5a8f94-kserve-bb96b6848-5hk8h" podUID="24ccc37d-027a-4097-8da5-04df417aef94" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Oct 30 21:02:38.852977 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:38.840686    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-65968b/distinct-2-w5-65968b-kserve-6d65c85f6c-dz5r7" podUID="d6751cf5-7df8-4ffa-b6c8-8b0c4f16ba2a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": context deadline exceeded"
Oct 30 21:02:38.852977 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:38.842863    4521 patch_prober.go:29] interesting pod/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-nvwt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body=
Oct 30 21:02:38.852977 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:38.842898    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-nvwt8" podUID="6cff2b61-dcbd-4450-b776-43eff57a4fd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused"
Oct 30 21:02:38.878670 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:38.865224    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c4b379/distinct-2-w1-c4b379-kserve-7999b785f5-97fz5" podUID="080a654d-a5bf-4845-a07f-cc8a7e6d0ace" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:02:38.878670 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:38.865911    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a8f94/distinct-2-w3-5a8f94-kserve-ffcc6449b-hv48k" podUID="bc0f5352-5c44-4c35-adf5-3fffab06b644" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": context deadline exceeded"
Oct 30 21:02:38.913538 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:38.913483    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a8f94/distinct-w3-5a8f94-kserve-5f8b544d59-mf95j" podUID="b8ac1cca-4ebd-474c-b32f-a3595a4cd847" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:02:38.925071 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:38.921849    4521 patch_prober.go:29] interesting pod/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-m9wn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Oct 30 21:02:38.925071 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:38.921907    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-m9wn4" podUID="9dc1356e-e12d-49f9-822e-f1ccf4fdaa3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Oct 30 21:02:38.925071 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:38.921985    4521 patch_prober.go:29] interesting pod/e2e-worker-w5-65968b-openshift-default-68896fcddc-sbpfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body=
Oct 30 21:02:38.925071 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:38.922006    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-sbpfk" podUID="7a5f91bb-7229-454f-a006-0d768f8879f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused"
Oct 30 21:02:38.925071 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:38.922306    4521 generic.go:356] "Generic (PLEG): container finished" podID="55990440-6bb2-4ce7-945f-6ada2906a855" containerID="f3f770366013af52bebfc6b7500a37d53693245530cfa7a6c52c4aa6ebbbbee0" exitCode=0
Oct 30 21:02:39.010815 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:39.006452    4521 patch_prober.go:29] interesting pod/e2e-worker-w0-4a0c5b-openshift-default-676865df99-jjmw4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:02:39.010815 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:39.006527    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-jjmw4" podUID="d91a4f29-950e-4370-92e0-8ec493b72620" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:02:39.228770 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:39.204094    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-39fb96-openshift-default-5f55489475-ctbdp" event={"ID":"55990440-6bb2-4ce7-945f-6ada2906a855","Type":"ContainerDied","Data":"f3f770366013af52bebfc6b7500a37d53693245530cfa7a6c52c4aa6ebbbbee0"}
Oct 30 21:02:39.416957 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:39.411878    4521 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-tqd8s"]
Oct 30 21:02:39.416957 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:39.411942    4521 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-hmwtj"]
Oct 30 21:02:39.416957 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:39.411972    4521 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-p55rm"]
Oct 30 21:02:39.416957 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:39.267485    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c2d1eb/premium-w4-c2d1eb-kserve-8696fbd9b9-mxqhm" podUID="4bb0f898-b77f-419d-bff4-6413e82ef51d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:02:39.423103 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:39.423062    4521 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-p55rm" podUID="baad1994-acb5-4aae-bb65-a5e8a5b8afdc" containerName="istio-proxy" containerID="cri-o://792caf04aadf9b69e35ec7d46269a59947e1107749f2465ea6a82633fae13abd" gracePeriod=30
Oct 30 21:02:39.423482 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:39.301426    4521 patch_prober.go:29] interesting pod/e2e-worker-w2-32a969-openshift-default-8945d87fc-p55rm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:02:39.423642 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:39.423617    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-p55rm" podUID="baad1994-acb5-4aae-bb65-a5e8a5b8afdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:02:39.423769 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:39.422833    4521 patch_prober.go:29] interesting pod/e2e-worker-w0-4a0c5b-openshift-default-676865df99-5tqcb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:02:39.423881 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:39.423860    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-5tqcb" podUID="11b85d0f-fa82-4861-9c1c-e47a1c1c308a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:02:39.431351 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:39.424111    4521 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-tqd8s" podUID="00b6adb9-3733-4ea4-9a26-f3b29991001a" containerName="istio-proxy" containerID="cri-o://e361ad15db1741dc7fcce5c213ec24d7f516d1ca35e3e8358688e2c51d1e3813" gracePeriod=30
Oct 30 21:02:39.552575 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:39.353670    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-39fb96-openshift-default-5f55489475-ctbdp"
Oct 30 21:02:39.584352 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:39.301745    4521 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-5a8f94/distinct-w3-5a8f94-kserve-5f8b544d59-mf95j" podUID="b8ac1cca-4ebd-474c-b32f-a3595a4cd847" containerName="main" containerID="cri-o://ec30aab6c0ae6126f6e198b644f2bcc5738d6823efd7ef36cca8326416a01eb2" gracePeriod=45
Oct 30 21:02:39.584835 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:39.442661    4521 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-hmwtj" podUID="f6f47707-ed2c-4860-a13a-b1a5d9d1bbe3" containerName="istio-proxy" containerID="cri-o://8b8bd567c67d21ef122a8fd6141d1c042d7bfca248ef665aa84d0e993110b661" gracePeriod=30
Oct 30 21:02:39.589346 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:39.339568    4521 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-5a8f94/premium-w3-5a8f94-kserve-bb96b6848-5hk8h" podUID="24ccc37d-027a-4097-8da5-04df417aef94" containerName="main" containerID="cri-o://47c33f2a9ce29670bd78c147b405d8d795a4c4b67ba92d3660b28bff3dd7026c" gracePeriod=45
Oct 30 21:02:39.598344 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:39.473686    4521 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-5a8f94/distinct-2-w3-5a8f94-kserve-ffcc6449b-hv48k" podUID="bc0f5352-5c44-4c35-adf5-3fffab06b644" containerName="main" containerID="cri-o://3e2e97d63e66c394818e2e06282f16a792a8be5acfb6030ebb53b47e8cd80d7d" gracePeriod=45
Oct 30 21:02:39.617373 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:39.473818    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c2d1eb/premium-w4-c2d1eb-kserve-8696fbd9b9-mxqhm"
Oct 30 21:02:39.625443 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:39.549190    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c4b379/premium-w1-c4b379-kserve-575f896d49-2n6gz" podUID="2aac3e8e-d631-4b9c-bdbb-3b4b1df133a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:02:39.650349 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:39.629398    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55990440-6bb2-4ce7-945f-6ada2906a855-istio-token\" (UniqueName: \"kubernetes.io/projected/55990440-6bb2-4ce7-945f-6ada2906a855-istio-token\") pod \"55990440-6bb2-4ce7-945f-6ada2906a855\" (UID: \"55990440-6bb2-4ce7-945f-6ada2906a855\") "
Oct 30 21:02:39.692593 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:39.692558    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55990440-6bb2-4ce7-945f-6ada2906a855-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55990440-6bb2-4ce7-945f-6ada2906a855-workload-socket\") pod \"55990440-6bb2-4ce7-945f-6ada2906a855\" (UID: \"55990440-6bb2-4ce7-945f-6ada2906a855\") "
Oct 30 21:02:39.695610 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:39.695581    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55990440-6bb2-4ce7-945f-6ada2906a855-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55990440-6bb2-4ce7-945f-6ada2906a855-istio-envoy\") pod \"55990440-6bb2-4ce7-945f-6ada2906a855\" (UID: \"55990440-6bb2-4ce7-945f-6ada2906a855\") "
Oct 30 21:02:39.701503 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:39.701474    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55990440-6bb2-4ce7-945f-6ada2906a855-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55990440-6bb2-4ce7-945f-6ada2906a855-credential-socket\") pod \"55990440-6bb2-4ce7-945f-6ada2906a855\" (UID: \"55990440-6bb2-4ce7-945f-6ada2906a855\") "
Oct 30 21:02:39.701808 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:39.701791    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55990440-6bb2-4ce7-945f-6ada2906a855-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55990440-6bb2-4ce7-945f-6ada2906a855-istio-data\") pod \"55990440-6bb2-4ce7-945f-6ada2906a855\" (UID: \"55990440-6bb2-4ce7-945f-6ada2906a855\") "
Oct 30 21:02:39.701985 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:39.701971    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55990440-6bb2-4ce7-945f-6ada2906a855-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55990440-6bb2-4ce7-945f-6ada2906a855-workload-certs\") pod \"55990440-6bb2-4ce7-945f-6ada2906a855\" (UID: \"55990440-6bb2-4ce7-945f-6ada2906a855\") "
Oct 30 21:02:39.702092 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:39.702079    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/55990440-6bb2-4ce7-945f-6ada2906a855-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55990440-6bb2-4ce7-945f-6ada2906a855-istio-podinfo\") pod \"55990440-6bb2-4ce7-945f-6ada2906a855\" (UID: \"55990440-6bb2-4ce7-945f-6ada2906a855\") "
Oct 30 21:02:39.702220 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:39.702204    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55990440-6bb2-4ce7-945f-6ada2906a855-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55990440-6bb2-4ce7-945f-6ada2906a855-istiod-ca-cert\") pod \"55990440-6bb2-4ce7-945f-6ada2906a855\" (UID: \"55990440-6bb2-4ce7-945f-6ada2906a855\") "
Oct 30 21:02:39.727502 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:39.727458    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55990440-6bb2-4ce7-945f-6ada2906a855-kube-api-access-ctk9j\" (UniqueName: \"kubernetes.io/projected/55990440-6bb2-4ce7-945f-6ada2906a855-kube-api-access-ctk9j\") pod \"55990440-6bb2-4ce7-945f-6ada2906a855\" (UID: \"55990440-6bb2-4ce7-945f-6ada2906a855\") "
Oct 30 21:02:39.728945 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:39.655022    4521 patch_prober.go:29] interesting pod/e2e-worker-w5-65968b-openshift-default-68896fcddc-9zp68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body=
Oct 30 21:02:39.729164 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:39.729132    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-9zp68" podUID="65e4dd7b-506f-4350-a82c-e35e61706595" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused"
Oct 30 21:02:39.766384 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:39.732655    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55990440-6bb2-4ce7-945f-6ada2906a855-istiod-ca-cert" pod "55990440-6bb2-4ce7-945f-6ada2906a855" (UID: "55990440-6bb2-4ce7-945f-6ada2906a855"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:02:39.766384 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:39.695462    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55990440-6bb2-4ce7-945f-6ada2906a855-workload-socket" pod "55990440-6bb2-4ce7-945f-6ada2906a855" (UID: "55990440-6bb2-4ce7-945f-6ada2906a855"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:02:39.766384 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:39.734131    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55990440-6bb2-4ce7-945f-6ada2906a855-istio-data" pod "55990440-6bb2-4ce7-945f-6ada2906a855" (UID: "55990440-6bb2-4ce7-945f-6ada2906a855"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:02:39.766384 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:39.747827    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55990440-6bb2-4ce7-945f-6ada2906a855-credential-socket" pod "55990440-6bb2-4ce7-945f-6ada2906a855" (UID: "55990440-6bb2-4ce7-945f-6ada2906a855"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:02:39.766384 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:39.754571    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55990440-6bb2-4ce7-945f-6ada2906a855-workload-certs" pod "55990440-6bb2-4ce7-945f-6ada2906a855" (UID: "55990440-6bb2-4ce7-945f-6ada2906a855"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:02:39.766384 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:39.764216    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a8f94/distinct-2-w3-5a8f94-kserve-ffcc6449b-hv48k" podUID="bc0f5352-5c44-4c35-adf5-3fffab06b644" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": read tcp 10.133.0.2:53760->10.133.0.82:8000: read: connection reset by peer"
Oct 30 21:02:39.767114 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:39.766645    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a8f94/distinct-2-w3-5a8f94-kserve-ffcc6449b-hv48k" podUID="bc0f5352-5c44-4c35-adf5-3fffab06b644" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": dial tcp 10.133.0.82:8000: connect: connection refused"
Oct 30 21:02:39.767114 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:39.766721    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a8f94/simulator-w3-5a8f94-kserve-77cc88cc89-n4sl6" podUID="1fa6900c-eee2-4f00-b0ec-248def48dfc4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:02:39.827406 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:39.824581    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55990440-6bb2-4ce7-945f-6ada2906a855-istio-envoy" pod "55990440-6bb2-4ce7-945f-6ada2906a855" (UID: "55990440-6bb2-4ce7-945f-6ada2906a855"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:02:39.840171 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:39.840120    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a8f94/premium-w3-5a8f94-kserve-bb96b6848-5hk8h" podUID="24ccc37d-027a-4097-8da5-04df417aef94" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:02:39.840825 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:39.840792    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-65968b/premium-w5-65968b-kserve-67977485f7-c9jpv" podUID="92b2dd65-8a83-436d-afa8-50e5f0d11fa0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 30 21:02:39.841293 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:39.841272    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-65968b/premium-w5-65968b-kserve-67977485f7-c9jpv"
Oct 30 21:02:39.842952 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:39.842928    4521 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55990440-6bb2-4ce7-945f-6ada2906a855-workload-socket\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:39.844695 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:39.844674    4521 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55990440-6bb2-4ce7-945f-6ada2906a855-istio-envoy\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:39.844799 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:39.844784    4521 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55990440-6bb2-4ce7-945f-6ada2906a855-credential-socket\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:39.844900 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:39.844888    4521 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55990440-6bb2-4ce7-945f-6ada2906a855-istio-data\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:39.844995 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:39.844984    4521 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55990440-6bb2-4ce7-945f-6ada2906a855-workload-certs\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:39.845084 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:39.845071    4521 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55990440-6bb2-4ce7-945f-6ada2906a855-istiod-ca-cert\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:39.845204 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:39.845182    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-65968b/distinct-2-w5-65968b-kserve-6d65c85f6c-dz5r7" podUID="d6751cf5-7df8-4ffa-b6c8-8b0c4f16ba2a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:02:39.890352 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:39.873901    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c4b379/distinct-2-w1-c4b379-kserve-7999b785f5-97fz5" podUID="080a654d-a5bf-4845-a07f-cc8a7e6d0ace" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:02:39.901366 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:39.891472    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c4b379/distinct-2-w1-c4b379-kserve-7999b785f5-97fz5"
Oct 30 21:02:39.929352 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:39.899481    4521 patch_prober.go:29] interesting pod/e2e-worker-w5-65968b-openshift-default-68896fcddc-sbpfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body=
Oct 30 21:02:40.113258 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:40.113163    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-sbpfk" podUID="7a5f91bb-7229-454f-a006-0d768f8879f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused"
Oct 30 21:02:40.256220 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:39.901551    4521 patch_prober.go:29] interesting pod/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-m9wn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Oct 30 21:02:40.281815 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:40.281770    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-m9wn4" podUID="9dc1356e-e12d-49f9-822e-f1ccf4fdaa3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Oct 30 21:02:40.282036 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:39.925775    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a8f94/distinct-w3-5a8f94-kserve-5f8b544d59-mf95j" podUID="b8ac1cca-4ebd-474c-b32f-a3595a4cd847" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded"
Oct 30 21:02:40.282446 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:39.909314    4521 patch_prober.go:29] interesting pod/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-nvwt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body=
Oct 30 21:02:40.282638 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:40.282612    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-nvwt8" podUID="6cff2b61-dcbd-4450-b776-43eff57a4fd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused"
Oct 30 21:02:40.282744 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:40.014473    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/55990440-6bb2-4ce7-945f-6ada2906a855-istio-podinfo" pod "55990440-6bb2-4ce7-945f-6ada2906a855" (UID: "55990440-6bb2-4ce7-945f-6ada2906a855"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:02:40.282828 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:40.234495    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55990440-6bb2-4ce7-945f-6ada2906a855-kube-api-access-ctk9j" pod "55990440-6bb2-4ce7-945f-6ada2906a855" (UID: "55990440-6bb2-4ce7-945f-6ada2906a855"). InnerVolumeSpecName "kube-api-access-ctk9j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:02:40.283048 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:40.283017    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a8f94/distinct-w3-5a8f94-kserve-5f8b544d59-mf95j" podUID="b8ac1cca-4ebd-474c-b32f-a3595a4cd847" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": dial tcp 10.133.0.71:8000: connect: connection refused"
Oct 30 21:02:40.283896 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:40.283872    4521 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctk9j\" (UniqueName: \"kubernetes.io/projected/55990440-6bb2-4ce7-945f-6ada2906a855-kube-api-access-ctk9j\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:40.284052 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:40.284029    4521 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55990440-6bb2-4ce7-945f-6ada2906a855-istio-podinfo\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:40.307690 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:40.307645    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55990440-6bb2-4ce7-945f-6ada2906a855-istio-token" pod "55990440-6bb2-4ce7-945f-6ada2906a855" (UID: "55990440-6bb2-4ce7-945f-6ada2906a855"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:02:40.392832 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:40.314758    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-39fb96-openshift-default-5f55489475-ctbdp" event={"ID":"55990440-6bb2-4ce7-945f-6ada2906a855","Type":"ContainerDied","Data":"4b3c8f3fca2a92a37a937714474ca054fde5bcb6876b18e37273c7c7eee57264"}
Oct 30 21:02:40.392832 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:40.314831    4521 scope.go:122] "RemoveContainer" containerID="f3f770366013af52bebfc6b7500a37d53693245530cfa7a6c52c4aa6ebbbbee0"
Oct 30 21:02:40.392832 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:40.315046    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-39fb96-openshift-default-5f55489475-ctbdp"
Oct 30 21:02:40.409733 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:40.407900    4521 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55990440-6bb2-4ce7-945f-6ada2906a855-istio-token\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:40.429503 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:40.429457    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c2d1eb/premium-w4-c2d1eb-kserve-8696fbd9b9-mxqhm" podUID="4bb0f898-b77f-419d-bff4-6413e82ef51d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded"
Oct 30 21:02:40.726409 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:40.710493    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c4b379/premium-w1-c4b379-kserve-575f896d49-2n6gz" podUID="2aac3e8e-d631-4b9c-bdbb-3b4b1df133a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:02:40.726409 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:40.711934    4521 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-65968b/premium-w5-65968b-kserve-67977485f7-c9jpv" podUID="92b2dd65-8a83-436d-afa8-50e5f0d11fa0" containerName="main" containerID="cri-o://f850ab3fba071258f1ebd3015d73e51d8c7a3c2027a8c8a5c70910ca96df1a37" gracePeriod=45
Oct 30 21:02:40.731295 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:40.731261    4521 patch_prober.go:29] interesting pod/e2e-worker-w5-65968b-openshift-default-68896fcddc-9zp68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body=
Oct 30 21:02:40.731612 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:40.731559    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-9zp68" podUID="65e4dd7b-506f-4350-a82c-e35e61706595" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused"
Oct 30 21:02:40.732485 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:40.732440    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a8f94/distinct-2-w3-5a8f94-kserve-ffcc6449b-hv48k" podUID="bc0f5352-5c44-4c35-adf5-3fffab06b644" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": dial tcp 10.133.0.82:8000: connect: connection refused"
Oct 30 21:02:40.780476 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:40.780429    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a8f94/simulator-w3-5a8f94-kserve-77cc88cc89-n4sl6" podUID="1fa6900c-eee2-4f00-b0ec-248def48dfc4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:02:40.869351 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:40.849771    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-65968b/distinct-2-w5-65968b-kserve-6d65c85f6c-dz5r7" podUID="d6751cf5-7df8-4ffa-b6c8-8b0c4f16ba2a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:02:40.869351 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:40.858970    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a8f94/premium-w3-5a8f94-kserve-bb96b6848-5hk8h" podUID="24ccc37d-027a-4097-8da5-04df417aef94" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Oct 30 21:02:40.895296 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:40.895264    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5a8f94/simulator-w3-5a8f94-kserve-77cc88cc89-n4sl6"
Oct 30 21:02:40.933916 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:40.933888    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-nvwt8"
Oct 30 21:02:40.934127 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:40.934116    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-m9wn4"
Oct 30 21:02:40.935129 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:40.895568    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c4b379/distinct-2-w1-c4b379-kserve-7999b785f5-97fz5" podUID="080a654d-a5bf-4845-a07f-cc8a7e6d0ace" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:02:40.935875 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:40.895602    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-65968b/premium-w5-65968b-kserve-67977485f7-c9jpv" podUID="92b2dd65-8a83-436d-afa8-50e5f0d11fa0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 30 21:02:40.936968 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:40.901045    4521 patch_prober.go:29] interesting pod/e2e-worker-w5-65968b-openshift-default-68896fcddc-sbpfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body=
Oct 30 21:02:40.937127 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:40.937104    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-sbpfk" podUID="7a5f91bb-7229-454f-a006-0d768f8879f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused"
Oct 30 21:02:40.954134 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:40.901111    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a8f94/premium-w3-5a8f94-kserve-bb96b6848-5hk8h" podUID="24ccc37d-027a-4097-8da5-04df417aef94" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": read tcp 10.133.0.2:53710->10.133.0.53:8000: read: connection reset by peer"
Oct 30 21:02:40.954475 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:40.949607    4521 patch_prober.go:29] interesting pod/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-m9wn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Oct 30 21:02:40.954647 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:40.954623    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-m9wn4" podUID="9dc1356e-e12d-49f9-822e-f1ccf4fdaa3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Oct 30 21:02:40.954742 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:40.950341    4521 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-65968b/distinct-2-w5-65968b-kserve-6d65c85f6c-dz5r7" podUID="d6751cf5-7df8-4ffa-b6c8-8b0c4f16ba2a" containerName="main" containerID="cri-o://7e134bd3e9ae8eb8c9af61c92d85f54bc8f6e034ab4f9fcbca50396793babf02" gracePeriod=44
Oct 30 21:02:41.155500 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:41.094485    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a8f94/premium-w3-5a8f94-kserve-bb96b6848-5hk8h" podUID="24ccc37d-027a-4097-8da5-04df417aef94" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": dial tcp 10.133.0.53:8000: connect: connection refused"
Oct 30 21:02:41.155500 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:41.094810    4521 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-5a8f94/simulator-w3-5a8f94-kserve-77cc88cc89-n4sl6" podUID="1fa6900c-eee2-4f00-b0ec-248def48dfc4" containerName="main" containerID="cri-o://155ef94dbc2d28a469b16f4b47f77b5b424a93c6c7b687308f435f40ee44d4cc" gracePeriod=44
Oct 30 21:02:41.155500 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:41.114596    4521 patch_prober.go:29] interesting pod/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-nvwt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body=
Oct 30 21:02:41.155500 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:41.114658    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-nvwt8" podUID="6cff2b61-dcbd-4450-b776-43eff57a4fd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused"
Oct 30 21:02:41.294030 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:41.271617    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a8f94/distinct-w3-5a8f94-kserve-5f8b544d59-mf95j" podUID="b8ac1cca-4ebd-474c-b32f-a3595a4cd847" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": dial tcp 10.133.0.71:8000: connect: connection refused"
Oct 30 21:02:41.436568 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:41.436522    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c2d1eb/premium-w4-c2d1eb-kserve-8696fbd9b9-mxqhm" podUID="4bb0f898-b77f-419d-bff4-6413e82ef51d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded"
Oct 30 21:02:41.602935 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:41.602842    4521 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-408918eb-f26js"]
Oct 30 21:02:41.604833 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:41.604801    4521 cpu_manager.go:414] "containerMap: removing container" podUID="9b9a4c83-9045-460b-bec6-8c2a6c3615a3" containerName="revoke-keys"
Oct 30 21:02:41.604985 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:41.604969    4521 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b9a4c83-9045-460b-bec6-8c2a6c3615a3" containerName="revoke-keys"
Oct 30 21:02:41.605114 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:41.605100    4521 cpu_manager.go:414] "containerMap: removing container" podUID="7bd27e7e-be97-4796-84dd-72f42b5bf42e" containerName="revoke-keys"
Oct 30 21:02:41.605192 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:41.605182    4521 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7bd27e7e-be97-4796-84dd-72f42b5bf42e" containerName="revoke-keys"
Oct 30 21:02:41.605286 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:41.605275    4521 cpu_manager.go:414] "containerMap: removing container" podUID="00e42758-77ce-4663-adfa-2202c0aae24a" containerName="main"
Oct 30 21:02:41.605404 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:41.605391    4521 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00e42758-77ce-4663-adfa-2202c0aae24a" containerName="main"
Oct 30 21:02:41.605498 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:41.605488    4521 cpu_manager.go:414] "containerMap: removing container" podUID="55990440-6bb2-4ce7-945f-6ada2906a855" containerName="istio-proxy"
Oct 30 21:02:41.605587 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:41.605576    4521 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55990440-6bb2-4ce7-945f-6ada2906a855" containerName="istio-proxy"
Oct 30 21:02:41.700569 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:41.633699    4521 cpu_manager.go:414] "containerMap: removing container" podUID="28e05ac8-5b4d-4085-bf8c-58793ca1c370" containerName="revoke-keys"
Oct 30 21:02:41.700569 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:41.633734    4521 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28e05ac8-5b4d-4085-bf8c-58793ca1c370" containerName="revoke-keys"
Oct 30 21:02:41.700569 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:41.634127    4521 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b9a4c83-9045-460b-bec6-8c2a6c3615a3" containerName="revoke-keys"
Oct 30 21:02:41.700569 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:41.634150    4521 memory_manager.go:365] "RemoveStaleState removing state" podUID="7bd27e7e-be97-4796-84dd-72f42b5bf42e" containerName="revoke-keys"
Oct 30 21:02:41.700569 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:41.634170    4521 memory_manager.go:365] "RemoveStaleState removing state" podUID="00e42758-77ce-4663-adfa-2202c0aae24a" containerName="main"
Oct 30 21:02:41.700569 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:41.634181    4521 memory_manager.go:365] "RemoveStaleState removing state" podUID="55990440-6bb2-4ce7-945f-6ada2906a855" containerName="istio-proxy"
Oct 30 21:02:41.700569 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:41.634209    4521 memory_manager.go:365] "RemoveStaleState removing state" podUID="28e05ac8-5b4d-4085-bf8c-58793ca1c370" containerName="revoke-keys"
Oct 30 21:02:41.700569 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:41.663752    4521 patch_prober.go:29] interesting pod/e2e-worker-w5-65968b-openshift-default-68896fcddc-9zp68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body=
Oct 30 21:02:41.700569 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:41.663821    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-9zp68" podUID="65e4dd7b-506f-4350-a82c-e35e61706595" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused"
Oct 30 21:02:41.700569 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:41.664382    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a8f94/distinct-2-w3-5a8f94-kserve-ffcc6449b-hv48k" podUID="bc0f5352-5c44-4c35-adf5-3fffab06b644" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": dial tcp 10.133.0.82:8000: connect: connection refused"
Oct 30 21:02:41.777346 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:41.717843    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c4b379/premium-w1-c4b379-kserve-575f896d49-2n6gz" podUID="2aac3e8e-d631-4b9c-bdbb-3b4b1df133a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:02:41.800348 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:41.795484    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a8f94/simulator-w3-5a8f94-kserve-77cc88cc89-n4sl6" podUID="1fa6900c-eee2-4f00-b0ec-248def48dfc4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 30 21:02:41.881173 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:41.825802    4521 patch_prober.go:29] interesting pod/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-nvwt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body=
Oct 30 21:02:41.881173 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:41.825886    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-nvwt8" podUID="6cff2b61-dcbd-4450-b776-43eff57a4fd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused"
Oct 30 21:02:41.881173 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:41.856932    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-39fb96-openshift-default-5f55489475-z6jxt"
Oct 30 21:02:41.881173 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:41.856998    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-9zp68"
Oct 30 21:02:41.881173 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:41.857029    4521 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-39fb96-openshift-default-5f55489475-ctbdp"]
Oct 30 21:02:41.881173 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:41.857874    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-408918eb-f26js"
Oct 30 21:02:41.881173 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:41.870967    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-65968b/distinct-2-w5-65968b-kserve-6d65c85f6c-dz5r7" podUID="d6751cf5-7df8-4ffa-b6c8-8b0c4f16ba2a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:02:41.881173 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:41.876251    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-65968b/distinct-2-w5-65968b-kserve-6d65c85f6c-dz5r7" podUID="d6751cf5-7df8-4ffa-b6c8-8b0c4f16ba2a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": dial tcp 10.133.0.93:8000: connect: connection refused"
Oct 30 21:02:41.881173 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:41.876591    4521 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:02:41.881173 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:41.876745    4521 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c45pw\""
Oct 30 21:02:42.025937 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:41.939311    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-65968b/premium-w5-65968b-kserve-67977485f7-c9jpv" podUID="92b2dd65-8a83-436d-afa8-50e5f0d11fa0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 30 21:02:42.025937 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:41.939747    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c4b379/distinct-2-w1-c4b379-kserve-7999b785f5-97fz5" podUID="080a654d-a5bf-4845-a07f-cc8a7e6d0ace" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": context deadline exceeded"
Oct 30 21:02:42.025937 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:41.949858    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-65968b/premium-w5-65968b-kserve-67977485f7-c9jpv" podUID="92b2dd65-8a83-436d-afa8-50e5f0d11fa0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": dial tcp 10.133.0.54:8000: connect: connection refused"
Oct 30 21:02:42.025937 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:41.952280    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-sbpfk"
Oct 30 21:02:42.025937 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:41.956696    4521 patch_prober.go:29] interesting pod/e2e-worker-w5-65968b-openshift-default-68896fcddc-sbpfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body=
Oct 30 21:02:42.025937 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:41.956758    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-sbpfk" podUID="7a5f91bb-7229-454f-a006-0d768f8879f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused"
Oct 30 21:02:42.025937 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:41.979349    4521 patch_prober.go:29] interesting pod/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-m9wn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Oct 30 21:02:42.025937 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:41.979400    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-m9wn4" podUID="9dc1356e-e12d-49f9-822e-f1ccf4fdaa3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Oct 30 21:02:42.025937 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:42.005619    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-65968b/premium-w5-65968b-kserve-67977485f7-c9jpv" podUID="92b2dd65-8a83-436d-afa8-50e5f0d11fa0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": dial tcp 10.133.0.54:8000: connect: connection refused"
Oct 30 21:02:42.110378 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:42.072358    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a8f94/premium-w3-5a8f94-kserve-bb96b6848-5hk8h" podUID="24ccc37d-027a-4097-8da5-04df417aef94" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": dial tcp 10.133.0.53:8000: connect: connection refused"
Oct 30 21:02:42.178671 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:42.110574    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-65968b/distinct-2-w5-65968b-kserve-6d65c85f6c-dz5r7" podUID="d6751cf5-7df8-4ffa-b6c8-8b0c4f16ba2a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": dial tcp 10.133.0.93:8000: connect: connection refused"
Oct 30 21:02:42.242210 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:42.211186    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-408918eb-f26js"]
Oct 30 21:02:42.242210 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:42.211370    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c4b379/distinct-2-w1-c4b379-kserve-7999b785f5-97fz5"
Oct 30 21:02:42.274386 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:42.273462    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a8f94/distinct-w3-5a8f94-kserve-5f8b544d59-mf95j" podUID="b8ac1cca-4ebd-474c-b32f-a3595a4cd847" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": dial tcp 10.133.0.71:8000: connect: connection refused"
Oct 30 21:02:42.333575 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:42.333263    4521 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-c4b379/premium-w1-c4b379-kserve-575f896d49-2n6gz" podUID="2aac3e8e-d631-4b9c-bdbb-3b4b1df133a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": context deadline exceeded"
Oct 30 21:02:42.443403 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:42.434967    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d10d07e-a60e-4c6a-b067-66234939fa57-maas-api-xks-ca\") pod \"maas-api-revoke-sub-408918eb-f26js\" (UID: \"7d10d07e-a60e-4c6a-b067-66234939fa57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-408918eb-f26js"
Oct 30 21:02:42.443403 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:42.435057    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d10d07e-a60e-4c6a-b067-66234939fa57-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-408918eb-f26js\" (UID: \"7d10d07e-a60e-4c6a-b067-66234939fa57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-408918eb-f26js"
Oct 30 21:02:42.443403 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:42.435302    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6znsz\" (UniqueName: \"kubernetes.io/projected/7d10d07e-a60e-4c6a-b067-66234939fa57-kube-api-access-6znsz\") pod \"maas-api-revoke-sub-408918eb-f26js\" (UID: \"7d10d07e-a60e-4c6a-b067-66234939fa57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-408918eb-f26js"
Oct 30 21:02:42.471277 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:42.445901    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c2d1eb/premium-w4-c2d1eb-kserve-8696fbd9b9-mxqhm" podUID="4bb0f898-b77f-419d-bff4-6413e82ef51d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:02:42.591348 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:42.579292    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6znsz\" (UniqueName: \"kubernetes.io/projected/7d10d07e-a60e-4c6a-b067-66234939fa57-kube-api-access-6znsz\") pod \"maas-api-revoke-sub-408918eb-f26js\" (UID: \"7d10d07e-a60e-4c6a-b067-66234939fa57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-408918eb-f26js"
Oct 30 21:02:42.613606 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:42.604648    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d10d07e-a60e-4c6a-b067-66234939fa57-maas-api-xks-ca\") pod \"maas-api-revoke-sub-408918eb-f26js\" (UID: \"7d10d07e-a60e-4c6a-b067-66234939fa57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-408918eb-f26js"
Oct 30 21:02:42.613606 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:42.604729    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d10d07e-a60e-4c6a-b067-66234939fa57-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-408918eb-f26js\" (UID: \"7d10d07e-a60e-4c6a-b067-66234939fa57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-408918eb-f26js"
Oct 30 21:02:42.613606 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:42.606095    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d10d07e-a60e-4c6a-b067-66234939fa57-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-408918eb-f26js\" (UID: \"7d10d07e-a60e-4c6a-b067-66234939fa57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-408918eb-f26js"
Oct 30 21:02:42.745991 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:42.687676    4521 patch_prober.go:29] interesting pod/e2e-worker-w5-65968b-openshift-default-68896fcddc-9zp68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body=
Oct 30 21:02:42.745991 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:42.687741    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-9zp68" podUID="65e4dd7b-506f-4350-a82c-e35e61706595" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused"
Oct 30 21:02:42.745991 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:42.688221    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a8f94/distinct-2-w3-5a8f94-kserve-ffcc6449b-hv48k" podUID="bc0f5352-5c44-4c35-adf5-3fffab06b644" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": dial tcp 10.133.0.82:8000: connect: connection refused"
Oct 30 21:02:42.745991 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:42.716425    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6znsz\" (UniqueName: \"kubernetes.io/projected/7d10d07e-a60e-4c6a-b067-66234939fa57-kube-api-access-6znsz\") pod \"maas-api-revoke-sub-408918eb-f26js\" (UID: \"7d10d07e-a60e-4c6a-b067-66234939fa57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-408918eb-f26js"
Oct 30 21:02:42.745991 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:42.743462    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c4b379/premium-w1-c4b379-kserve-575f896d49-2n6gz" podUID="2aac3e8e-d631-4b9c-bdbb-3b4b1df133a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:02:42.831361 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:42.808460    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a8f94/simulator-w3-5a8f94-kserve-77cc88cc89-n4sl6" podUID="1fa6900c-eee2-4f00-b0ec-248def48dfc4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:02:42.920018 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:42.919958    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a8f94/simulator-w3-5a8f94-kserve-77cc88cc89-n4sl6" podUID="1fa6900c-eee2-4f00-b0ec-248def48dfc4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": read tcp 10.133.0.2:55980->10.133.0.51:8000: read: connection reset by peer"
Oct 30 21:02:42.938908 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:42.938871    4521 patch_prober.go:29] interesting pod/e2e-worker-w5-65968b-openshift-default-68896fcddc-sbpfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body=
Oct 30 21:02:43.149409 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:43.078598    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-sbpfk" podUID="7a5f91bb-7229-454f-a006-0d768f8879f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused"
Oct 30 21:02:43.149409 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:43.078425    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a8f94/simulator-w3-5a8f94-kserve-77cc88cc89-n4sl6" podUID="1fa6900c-eee2-4f00-b0ec-248def48dfc4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": dial tcp 10.133.0.51:8000: connect: connection refused"
Oct 30 21:02:43.149409 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:43.078368    4521 patch_prober.go:29] interesting pod/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-m9wn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Oct 30 21:02:43.149409 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:43.078995    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-m9wn4" podUID="9dc1356e-e12d-49f9-822e-f1ccf4fdaa3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Oct 30 21:02:43.151468 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:43.151426    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a8f94/premium-w3-5a8f94-kserve-bb96b6848-5hk8h" podUID="24ccc37d-027a-4097-8da5-04df417aef94" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": dial tcp 10.133.0.53:8000: connect: connection refused"
Oct 30 21:02:43.151895 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:43.151867    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-65968b/premium-w5-65968b-kserve-67977485f7-c9jpv" podUID="92b2dd65-8a83-436d-afa8-50e5f0d11fa0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": dial tcp 10.133.0.54:8000: connect: connection refused"
Oct 30 21:02:43.166544 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:43.152210    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-65968b/distinct-2-w5-65968b-kserve-6d65c85f6c-dz5r7" podUID="d6751cf5-7df8-4ffa-b6c8-8b0c4f16ba2a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": dial tcp 10.133.0.93:8000: connect: connection refused"
Oct 30 21:02:43.175550 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:43.168156    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d10d07e-a60e-4c6a-b067-66234939fa57-maas-api-xks-ca\") pod \"maas-api-revoke-sub-408918eb-f26js\" (UID: \"7d10d07e-a60e-4c6a-b067-66234939fa57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-408918eb-f26js"
Oct 30 21:02:43.200486 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:43.190937    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-408918eb-f26js"
Oct 30 21:02:43.291260 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:43.277724    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a8f94/distinct-w3-5a8f94-kserve-5f8b544d59-mf95j" podUID="b8ac1cca-4ebd-474c-b32f-a3595a4cd847" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": dial tcp 10.133.0.71:8000: connect: connection refused"
Oct 30 21:02:43.390702 ip-10-0-131-125 kubenswrapper[4521]: E1030 21:02:43.390647    4521 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55990440_6bb2_4ce7_945f_6ada2906a855.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbc0f5352_5c44_4c35_adf5_3fffab06b644.slice/crio-3e2e97d63e66c394818e2e06282f16a792a8be5acfb6030ebb53b47e8cd80d7d.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod24ccc37d_027a_4097_8da5_04df417aef94.slice/crio-47c33f2a9ce29670bd78c147b405d8d795a4c4b67ba92d3660b28bff3dd7026c.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55990440_6bb2_4ce7_945f_6ada2906a855.slice/crio-4b3c8f3fca2a92a37a937714474ca054fde5bcb6876b18e37273c7c7eee57264\": RecentStats: unable to find data in memory cache]"
Oct 30 21:02:43.454542 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:43.454493    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c2d1eb/premium-w4-c2d1eb-kserve-8696fbd9b9-mxqhm" podUID="4bb0f898-b77f-419d-bff4-6413e82ef51d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded"
Oct 30 21:02:43.555818 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:43.549875    4521 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-39fb96-openshift-default-5f55489475-ctbdp"]
Oct 30 21:02:43.671569 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:43.671532    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a8f94/distinct-2-w3-5a8f94-kserve-ffcc6449b-hv48k" podUID="bc0f5352-5c44-4c35-adf5-3fffab06b644" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": dial tcp 10.133.0.82:8000: connect: connection refused"
Oct 30 21:02:43.719423 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:43.718540    4521 patch_prober.go:29] interesting pod/e2e-worker-w5-65968b-openshift-default-68896fcddc-9zp68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body=
Oct 30 21:02:43.719423 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:43.718609    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-9zp68" podUID="65e4dd7b-506f-4350-a82c-e35e61706595" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused"
Oct 30 21:02:43.760247 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:43.758472    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c4b379/premium-w1-c4b379-kserve-575f896d49-2n6gz" podUID="2aac3e8e-d631-4b9c-bdbb-3b4b1df133a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:02:43.883302 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:43.882911    4521 patch_prober.go:29] interesting pod/e2e-worker-w5-65968b-openshift-default-68896fcddc-sbpfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body=
Oct 30 21:02:43.883302 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:43.882970    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-sbpfk" podUID="7a5f91bb-7229-454f-a006-0d768f8879f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused"
Oct 30 21:02:43.900745 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:43.900705    4521 patch_prober.go:29] interesting pod/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-m9wn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Oct 30 21:02:43.901266 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:43.900956    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-m9wn4" podUID="9dc1356e-e12d-49f9-822e-f1ccf4fdaa3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Oct 30 21:02:44.009402 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:43.976928    4521 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-617df4d2-t4ck6"]
Oct 30 21:02:44.040713 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:44.036348    4521 generic.go:356] "Generic (PLEG): container finished" podID="bc0f5352-5c44-4c35-adf5-3fffab06b644" containerID="3e2e97d63e66c394818e2e06282f16a792a8be5acfb6030ebb53b47e8cd80d7d" exitCode=0
Oct 30 21:02:44.057196 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:44.042063    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c4b379/premium-w1-c4b379-kserve-575f896d49-2n6gz"
Oct 30 21:02:44.057196 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:44.042097    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-5a8f94/distinct-2-w3-5a8f94-kserve-ffcc6449b-hv48k" event={"ID":"bc0f5352-5c44-4c35-adf5-3fffab06b644","Type":"ContainerDied","Data":"3e2e97d63e66c394818e2e06282f16a792a8be5acfb6030ebb53b47e8cd80d7d"}
Oct 30 21:02:44.057196 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:44.043064    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-617df4d2-t4ck6"
Oct 30 21:02:44.091346 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:44.071624    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a8f94/premium-w3-5a8f94-kserve-bb96b6848-5hk8h" podUID="24ccc37d-027a-4097-8da5-04df417aef94" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": dial tcp 10.133.0.53:8000: connect: connection refused"
Oct 30 21:02:44.093163 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:44.092916    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-65968b/premium-w5-65968b-kserve-67977485f7-c9jpv" podUID="92b2dd65-8a83-436d-afa8-50e5f0d11fa0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": dial tcp 10.133.0.54:8000: connect: connection refused"
Oct 30 21:02:44.114108 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:44.104502    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-65968b/distinct-2-w5-65968b-kserve-6d65c85f6c-dz5r7" podUID="d6751cf5-7df8-4ffa-b6c8-8b0c4f16ba2a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": dial tcp 10.133.0.93:8000: connect: connection refused"
Oct 30 21:02:44.155447 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:44.133135    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a8f94/simulator-w3-5a8f94-kserve-77cc88cc89-n4sl6" podUID="1fa6900c-eee2-4f00-b0ec-248def48dfc4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": dial tcp 10.133.0.51:8000: connect: connection refused"
Oct 30 21:02:44.158543 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:44.156802    4521 generic.go:356] "Generic (PLEG): container finished" podID="1fa6900c-eee2-4f00-b0ec-248def48dfc4" containerID="155ef94dbc2d28a469b16f4b47f77b5b424a93c6c7b687308f435f40ee44d4cc" exitCode=0
Oct 30 21:02:44.158543 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:44.156895    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-5a8f94/simulator-w3-5a8f94-kserve-77cc88cc89-n4sl6" event={"ID":"1fa6900c-eee2-4f00-b0ec-248def48dfc4","Type":"ContainerDied","Data":"155ef94dbc2d28a469b16f4b47f77b5b424a93c6c7b687308f435f40ee44d4cc"}
Oct 30 21:02:44.326911 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:44.228350    4521 patch_prober.go:29] interesting pod/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-nvwt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:02:44.326911 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:44.228409    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-nvwt8" podUID="6cff2b61-dcbd-4450-b776-43eff57a4fd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:02:44.326911 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:44.239304    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75d8f639-8cee-4be3-af28-eb7886627983-maas-api-xks-ca\") pod \"maas-api-revoke-sub-617df4d2-t4ck6\" (UID: \"75d8f639-8cee-4be3-af28-eb7886627983\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-617df4d2-t4ck6"
Oct 30 21:02:44.326911 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:44.242555    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55bth\" (UniqueName: \"kubernetes.io/projected/75d8f639-8cee-4be3-af28-eb7886627983-kube-api-access-55bth\") pod \"maas-api-revoke-sub-617df4d2-t4ck6\" (UID: \"75d8f639-8cee-4be3-af28-eb7886627983\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-617df4d2-t4ck6"
Oct 30 21:02:44.326911 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:44.242667    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75d8f639-8cee-4be3-af28-eb7886627983-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-617df4d2-t4ck6\" (UID: \"75d8f639-8cee-4be3-af28-eb7886627983\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-617df4d2-t4ck6"
Oct 30 21:02:44.326911 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:44.313487    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a8f94/distinct-w3-5a8f94-kserve-5f8b544d59-mf95j" podUID="b8ac1cca-4ebd-474c-b32f-a3595a4cd847" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": dial tcp 10.133.0.71:8000: connect: connection refused"
Oct 30 21:02:44.342898 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:44.342862    4521 generic.go:356] "Generic (PLEG): container finished" podID="92b2dd65-8a83-436d-afa8-50e5f0d11fa0" containerID="f850ab3fba071258f1ebd3015d73e51d8c7a3c2027a8c8a5c70910ca96df1a37" exitCode=0
Oct 30 21:02:44.343237 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:44.343203    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-65968b/premium-w5-65968b-kserve-67977485f7-c9jpv" event={"ID":"92b2dd65-8a83-436d-afa8-50e5f0d11fa0","Type":"ContainerDied","Data":"f850ab3fba071258f1ebd3015d73e51d8c7a3c2027a8c8a5c70910ca96df1a37"}
Oct 30 21:02:44.376412 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:44.373864    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75d8f639-8cee-4be3-af28-eb7886627983-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-617df4d2-t4ck6\" (UID: \"75d8f639-8cee-4be3-af28-eb7886627983\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-617df4d2-t4ck6"
Oct 30 21:02:44.376412 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:44.374119    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75d8f639-8cee-4be3-af28-eb7886627983-maas-api-xks-ca\") pod \"maas-api-revoke-sub-617df4d2-t4ck6\" (UID: \"75d8f639-8cee-4be3-af28-eb7886627983\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-617df4d2-t4ck6"
Oct 30 21:02:44.376412 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:44.374220    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55bth\" (UniqueName: \"kubernetes.io/projected/75d8f639-8cee-4be3-af28-eb7886627983-kube-api-access-55bth\") pod \"maas-api-revoke-sub-617df4d2-t4ck6\" (UID: \"75d8f639-8cee-4be3-af28-eb7886627983\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-617df4d2-t4ck6"
Oct 30 21:02:44.376412 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:44.375448    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75d8f639-8cee-4be3-af28-eb7886627983-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-617df4d2-t4ck6\" (UID: \"75d8f639-8cee-4be3-af28-eb7886627983\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-617df4d2-t4ck6"
Oct 30 21:02:44.541348 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:44.539561    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c2d1eb/premium-w4-c2d1eb-kserve-8696fbd9b9-mxqhm" podUID="4bb0f898-b77f-419d-bff4-6413e82ef51d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded"
Oct 30 21:02:44.575843 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:44.550209    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-617df4d2-t4ck6"]
Oct 30 21:02:44.646484 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:44.637661    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75d8f639-8cee-4be3-af28-eb7886627983-maas-api-xks-ca\") pod \"maas-api-revoke-sub-617df4d2-t4ck6\" (UID: \"75d8f639-8cee-4be3-af28-eb7886627983\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-617df4d2-t4ck6"
Oct 30 21:02:44.807234 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:44.710531    4521 patch_prober.go:29] interesting pod/e2e-worker-w5-65968b-openshift-default-68896fcddc-9zp68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body=
Oct 30 21:02:44.876784 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:44.872509    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-9zp68" podUID="65e4dd7b-506f-4350-a82c-e35e61706595" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused"
Oct 30 21:02:44.876784 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:44.825450    4521 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55990440-6bb2-4ce7-945f-6ada2906a855" path="/var/lib/kubelet/pods/55990440-6bb2-4ce7-945f-6ada2906a855/volumes"
Oct 30 21:02:44.932678 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:44.893562    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a8f94/distinct-2-w3-5a8f94-kserve-ffcc6449b-hv48k" podUID="bc0f5352-5c44-4c35-adf5-3fffab06b644" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": dial tcp 10.133.0.82:8000: connect: connection refused"
Oct 30 21:02:44.970719 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:44.945419    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55bth\" (UniqueName: \"kubernetes.io/projected/75d8f639-8cee-4be3-af28-eb7886627983-kube-api-access-55bth\") pod \"maas-api-revoke-sub-617df4d2-t4ck6\" (UID: \"75d8f639-8cee-4be3-af28-eb7886627983\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-617df4d2-t4ck6"
Oct 30 21:02:44.970719 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:44.969394    4521 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-72ee85f3-2hq26"]
Oct 30 21:02:45.090935 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:45.088733    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c4b379/distinct-2-w1-c4b379-kserve-7999b785f5-97fz5" podUID="080a654d-a5bf-4845-a07f-cc8a7e6d0ace" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:02:45.127436 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:45.110754    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a8f94/premium-w3-5a8f94-kserve-bb96b6848-5hk8h" podUID="24ccc37d-027a-4097-8da5-04df417aef94" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": dial tcp 10.133.0.53:8000: connect: connection refused"
Oct 30 21:02:45.127436 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:45.110876    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a8f94/simulator-w3-5a8f94-kserve-77cc88cc89-n4sl6" podUID="1fa6900c-eee2-4f00-b0ec-248def48dfc4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": dial tcp 10.133.0.51:8000: connect: connection refused"
Oct 30 21:02:45.127436 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:45.110952    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-65968b/distinct-2-w5-65968b-kserve-6d65c85f6c-dz5r7" podUID="d6751cf5-7df8-4ffa-b6c8-8b0c4f16ba2a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": dial tcp 10.133.0.93:8000: connect: connection refused"
Oct 30 21:02:45.127436 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:45.111362    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-617df4d2-t4ck6"
Oct 30 21:02:45.127436 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:45.114104    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-65968b/premium-w5-65968b-kserve-67977485f7-c9jpv" podUID="92b2dd65-8a83-436d-afa8-50e5f0d11fa0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": dial tcp 10.133.0.54:8000: connect: connection refused"
Oct 30 21:02:45.127436 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:45.114201    4521 patch_prober.go:29] interesting pod/e2e-worker-w5-65968b-openshift-default-68896fcddc-sbpfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body=
Oct 30 21:02:45.127436 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:45.114227    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-sbpfk" podUID="7a5f91bb-7229-454f-a006-0d768f8879f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused"
Oct 30 21:02:45.127436 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:45.114714    4521 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-c2d1eb/premium-w4-c2d1eb-kserve-8696fbd9b9-mxqhm" podUID="4bb0f898-b77f-419d-bff4-6413e82ef51d" containerName="main" containerID="cri-o://cf06834ad798b61733cac66d7babac65e2e8816c2cc87cdfd0eba97e9cd9d770" gracePeriod=45
Oct 30 21:02:45.133722 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:45.133555    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-72ee85f3-2hq26"]
Oct 30 21:02:45.133983 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:45.133701    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72ee85f3-2hq26"
Oct 30 21:02:45.166085 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:45.165975    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-408918eb-f26js"]
Oct 30 21:02:45.280899 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:45.279564    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a8f94/distinct-w3-5a8f94-kserve-5f8b544d59-mf95j" podUID="b8ac1cca-4ebd-474c-b32f-a3595a4cd847" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": dial tcp 10.133.0.71:8000: connect: connection refused"
Oct 30 21:02:45.280899 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:45.279821    4521 patch_prober.go:29] interesting pod/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-nvwt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:02:45.280899 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:45.279861    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-nvwt8" podUID="6cff2b61-dcbd-4450-b776-43eff57a4fd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:02:45.281476 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:45.281452    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c2d1eb/premium-w4-c2d1eb-kserve-8696fbd9b9-mxqhm" podUID="4bb0f898-b77f-419d-bff4-6413e82ef51d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": EOF"
Oct 30 21:02:45.282371 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:45.282311    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c2d1eb/premium-w4-c2d1eb-kserve-8696fbd9b9-mxqhm" podUID="4bb0f898-b77f-419d-bff4-6413e82ef51d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": dial tcp 10.133.0.56:8000: connect: connection refused"
Oct 30 21:02:45.285523 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:45.284862    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c4b379/premium-w1-c4b379-kserve-575f896d49-2n6gz" podUID="2aac3e8e-d631-4b9c-bdbb-3b4b1df133a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:02:45.368129 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:45.365917    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e857e76-3d48-4e5b-a8b5-830cfac13f90-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-72ee85f3-2hq26\" (UID: \"1e857e76-3d48-4e5b-a8b5-830cfac13f90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72ee85f3-2hq26"
Oct 30 21:02:45.368129 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:45.366062    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spnvs\" (UniqueName: \"kubernetes.io/projected/1e857e76-3d48-4e5b-a8b5-830cfac13f90-kube-api-access-spnvs\") pod \"maas-api-revoke-sub-72ee85f3-2hq26\" (UID: \"1e857e76-3d48-4e5b-a8b5-830cfac13f90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72ee85f3-2hq26"
Oct 30 21:02:45.368129 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:45.366149    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e857e76-3d48-4e5b-a8b5-830cfac13f90-maas-api-xks-ca\") pod \"maas-api-revoke-sub-72ee85f3-2hq26\" (UID: \"1e857e76-3d48-4e5b-a8b5-830cfac13f90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72ee85f3-2hq26"
Oct 30 21:02:45.470857 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:45.470094    4521 generic.go:356] "Generic (PLEG): container finished" podID="d6751cf5-7df8-4ffa-b6c8-8b0c4f16ba2a" containerID="7e134bd3e9ae8eb8c9af61c92d85f54bc8f6e034ab4f9fcbca50396793babf02" exitCode=0
Oct 30 21:02:45.470857 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:45.470408    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-65968b/distinct-2-w5-65968b-kserve-6d65c85f6c-dz5r7" event={"ID":"d6751cf5-7df8-4ffa-b6c8-8b0c4f16ba2a","Type":"ContainerDied","Data":"7e134bd3e9ae8eb8c9af61c92d85f54bc8f6e034ab4f9fcbca50396793babf02"}
Oct 30 21:02:45.485588 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:45.485477    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-408918eb-f26js" event={"ID":"7d10d07e-a60e-4c6a-b067-66234939fa57","Type":"ContainerStarted","Data":"967aca5d2c6e3b1db0100f3d227c95655e0f7db62296606e1bb1560e12796b35"}
Oct 30 21:02:45.625437 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:45.623180    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spnvs\" (UniqueName: \"kubernetes.io/projected/1e857e76-3d48-4e5b-a8b5-830cfac13f90-kube-api-access-spnvs\") pod \"maas-api-revoke-sub-72ee85f3-2hq26\" (UID: \"1e857e76-3d48-4e5b-a8b5-830cfac13f90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72ee85f3-2hq26"
Oct 30 21:02:45.625437 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:45.623303    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e857e76-3d48-4e5b-a8b5-830cfac13f90-maas-api-xks-ca\") pod \"maas-api-revoke-sub-72ee85f3-2hq26\" (UID: \"1e857e76-3d48-4e5b-a8b5-830cfac13f90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72ee85f3-2hq26"
Oct 30 21:02:45.625437 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:45.623446    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e857e76-3d48-4e5b-a8b5-830cfac13f90-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-72ee85f3-2hq26\" (UID: \"1e857e76-3d48-4e5b-a8b5-830cfac13f90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72ee85f3-2hq26"
Oct 30 21:02:45.628639 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:45.627641    4521 generic.go:356] "Generic (PLEG): container finished" podID="24ccc37d-027a-4097-8da5-04df417aef94" containerID="47c33f2a9ce29670bd78c147b405d8d795a4c4b67ba92d3660b28bff3dd7026c" exitCode=0
Oct 30 21:02:45.628639 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:45.627923    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-5a8f94/premium-w3-5a8f94-kserve-bb96b6848-5hk8h" event={"ID":"24ccc37d-027a-4097-8da5-04df417aef94","Type":"ContainerDied","Data":"47c33f2a9ce29670bd78c147b405d8d795a4c4b67ba92d3660b28bff3dd7026c"}
Oct 30 21:02:45.633092 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:45.629696    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e857e76-3d48-4e5b-a8b5-830cfac13f90-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-72ee85f3-2hq26\" (UID: \"1e857e76-3d48-4e5b-a8b5-830cfac13f90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72ee85f3-2hq26"
Oct 30 21:02:45.677961 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:45.677923    4521 generic.go:356] "Generic (PLEG): container finished" podID="4bb0f898-b77f-419d-bff4-6413e82ef51d" containerID="cf06834ad798b61733cac66d7babac65e2e8816c2cc87cdfd0eba97e9cd9d770" exitCode=0
Oct 30 21:02:45.678395 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:45.678365    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c2d1eb/premium-w4-c2d1eb-kserve-8696fbd9b9-mxqhm" event={"ID":"4bb0f898-b77f-419d-bff4-6413e82ef51d","Type":"ContainerDied","Data":"cf06834ad798b61733cac66d7babac65e2e8816c2cc87cdfd0eba97e9cd9d770"}
Oct 30 21:02:45.679788 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:45.679522    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a8f94/distinct-2-w3-5a8f94-kserve-ffcc6449b-hv48k" podUID="bc0f5352-5c44-4c35-adf5-3fffab06b644" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": dial tcp 10.133.0.82:8000: connect: connection refused"
Oct 30 21:02:45.682580 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:45.682554    4521 patch_prober.go:29] interesting pod/e2e-worker-w5-65968b-openshift-default-68896fcddc-9zp68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body=
Oct 30 21:02:45.708873 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:45.708831    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-9zp68" podUID="65e4dd7b-506f-4350-a82c-e35e61706595" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused"
Oct 30 21:02:45.712788 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:45.712740    4521 generic.go:356] "Generic (PLEG): container finished" podID="b8ac1cca-4ebd-474c-b32f-a3595a4cd847" containerID="ec30aab6c0ae6126f6e198b644f2bcc5738d6823efd7ef36cca8326416a01eb2" exitCode=0
Oct 30 21:02:45.713059 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:45.713038    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-5a8f94/distinct-w3-5a8f94-kserve-5f8b544d59-mf95j" event={"ID":"b8ac1cca-4ebd-474c-b32f-a3595a4cd847","Type":"ContainerDied","Data":"ec30aab6c0ae6126f6e198b644f2bcc5738d6823efd7ef36cca8326416a01eb2"}
Oct 30 21:02:45.910412 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:45.883017    4521 patch_prober.go:29] interesting pod/e2e-worker-w5-65968b-openshift-default-68896fcddc-sbpfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body=
Oct 30 21:02:45.910412 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:45.883103    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-sbpfk" podUID="7a5f91bb-7229-454f-a006-0d768f8879f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused"
Oct 30 21:02:45.910412 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:45.884933    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e857e76-3d48-4e5b-a8b5-830cfac13f90-maas-api-xks-ca\") pod \"maas-api-revoke-sub-72ee85f3-2hq26\" (UID: \"1e857e76-3d48-4e5b-a8b5-830cfac13f90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72ee85f3-2hq26"
Oct 30 21:02:45.910412 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:45.907555    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spnvs\" (UniqueName: \"kubernetes.io/projected/1e857e76-3d48-4e5b-a8b5-830cfac13f90-kube-api-access-spnvs\") pod \"maas-api-revoke-sub-72ee85f3-2hq26\" (UID: \"1e857e76-3d48-4e5b-a8b5-830cfac13f90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72ee85f3-2hq26"
Oct 30 21:02:45.930179 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:45.929494    4521 patch_prober.go:29] interesting pod/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-m9wn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:02:45.930179 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:45.929595    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-m9wn4" podUID="9dc1356e-e12d-49f9-822e-f1ccf4fdaa3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:02:46.020352 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:46.019648    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-65968b/premium-w5-65968b-kserve-67977485f7-c9jpv" podUID="92b2dd65-8a83-436d-afa8-50e5f0d11fa0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": dial tcp 10.133.0.54:8000: connect: connection refused"
Oct 30 21:02:46.080524 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:46.079882    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a8f94/simulator-w3-5a8f94-kserve-77cc88cc89-n4sl6" podUID="1fa6900c-eee2-4f00-b0ec-248def48dfc4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": dial tcp 10.133.0.51:8000: connect: connection refused"
Oct 30 21:02:46.081795 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:46.080991    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-65968b/distinct-2-w5-65968b-kserve-6d65c85f6c-dz5r7" podUID="d6751cf5-7df8-4ffa-b6c8-8b0c4f16ba2a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": dial tcp 10.133.0.93:8000: connect: connection refused"
Oct 30 21:02:46.083420 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:46.083102    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72ee85f3-2hq26"
Oct 30 21:02:46.336664 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:46.332182    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a8f94/distinct-w3-5a8f94-kserve-5f8b544d59-mf95j" podUID="b8ac1cca-4ebd-474c-b32f-a3595a4cd847" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": dial tcp 10.133.0.71:8000: connect: connection refused"
Oct 30 21:02:46.407275 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:46.405973    4521 patch_prober.go:29] interesting pod/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-nvwt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:02:46.407275 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:46.406042    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-nvwt8" podUID="6cff2b61-dcbd-4450-b776-43eff57a4fd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:02:46.504821 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:46.484538    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-5a8f94/premium-w3-5a8f94-kserve-bb96b6848-5hk8h"
Oct 30 21:02:46.741477 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:46.713081    4521 patch_prober.go:29] interesting pod/e2e-worker-w5-65968b-openshift-default-68896fcddc-9zp68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body=
Oct 30 21:02:46.741477 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:46.713142    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-9zp68" podUID="65e4dd7b-506f-4350-a82c-e35e61706595" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused"
Oct 30 21:02:46.741477 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:46.713505    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a8f94/distinct-2-w3-5a8f94-kserve-ffcc6449b-hv48k" podUID="bc0f5352-5c44-4c35-adf5-3fffab06b644" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": dial tcp 10.133.0.82:8000: connect: connection refused"
Oct 30 21:02:46.741477 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:46.721778    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24ccc37d-027a-4097-8da5-04df417aef94-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24ccc37d-027a-4097-8da5-04df417aef94-model-cache\") pod \"24ccc37d-027a-4097-8da5-04df417aef94\" (UID: \"24ccc37d-027a-4097-8da5-04df417aef94\") "
Oct 30 21:02:46.741477 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:46.721856    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24ccc37d-027a-4097-8da5-04df417aef94-kube-api-access-dkrx6\" (UniqueName: \"kubernetes.io/projected/24ccc37d-027a-4097-8da5-04df417aef94-kube-api-access-dkrx6\") pod \"24ccc37d-027a-4097-8da5-04df417aef94\" (UID: \"24ccc37d-027a-4097-8da5-04df417aef94\") "
Oct 30 21:02:46.741477 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:46.721919    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24ccc37d-027a-4097-8da5-04df417aef94-dshm\" (UniqueName: \"kubernetes.io/empty-dir/24ccc37d-027a-4097-8da5-04df417aef94-dshm\") pod \"24ccc37d-027a-4097-8da5-04df417aef94\" (UID: \"24ccc37d-027a-4097-8da5-04df417aef94\") "
Oct 30 21:02:46.741477 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:46.721989    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24ccc37d-027a-4097-8da5-04df417aef94-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24ccc37d-027a-4097-8da5-04df417aef94-tmp-dir\") pod \"24ccc37d-027a-4097-8da5-04df417aef94\" (UID: \"24ccc37d-027a-4097-8da5-04df417aef94\") "
Oct 30 21:02:46.741477 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:46.732376    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24ccc37d-027a-4097-8da5-04df417aef94-home\" (UniqueName: \"kubernetes.io/empty-dir/24ccc37d-027a-4097-8da5-04df417aef94-home\") pod \"24ccc37d-027a-4097-8da5-04df417aef94\" (UID: \"24ccc37d-027a-4097-8da5-04df417aef94\") "
Oct 30 21:02:46.741477 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:46.732620    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/24ccc37d-027a-4097-8da5-04df417aef94-tls-certs\" (UniqueName: \"kubernetes.io/secret/24ccc37d-027a-4097-8da5-04df417aef94-tls-certs\") pod \"24ccc37d-027a-4097-8da5-04df417aef94\" (UID: \"24ccc37d-027a-4097-8da5-04df417aef94\") "
Oct 30 21:02:46.802365 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:46.791089    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24ccc37d-027a-4097-8da5-04df417aef94-home" pod "24ccc37d-027a-4097-8da5-04df417aef94" (UID: "24ccc37d-027a-4097-8da5-04df417aef94"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:02:46.802365 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:46.793667    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24ccc37d-027a-4097-8da5-04df417aef94-model-cache" pod "24ccc37d-027a-4097-8da5-04df417aef94" (UID: "24ccc37d-027a-4097-8da5-04df417aef94"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:02:46.802365 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:46.798975    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24ccc37d-027a-4097-8da5-04df417aef94-tmp-dir" pod "24ccc37d-027a-4097-8da5-04df417aef94" (UID: "24ccc37d-027a-4097-8da5-04df417aef94"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:02:46.814145 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:46.814083    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24ccc37d-027a-4097-8da5-04df417aef94-kube-api-access-dkrx6" pod "24ccc37d-027a-4097-8da5-04df417aef94" (UID: "24ccc37d-027a-4097-8da5-04df417aef94"). InnerVolumeSpecName "kube-api-access-dkrx6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:02:46.835635 ip-10-0-131-125 kubenswrapper[4521]: E1030 21:02:46.835594    4521 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lkn2k" containerName="extractor"
Oct 30 21:02:46.914481 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:46.914364    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24ccc37d-027a-4097-8da5-04df417aef94-tls-certs" pod "24ccc37d-027a-4097-8da5-04df417aef94" (UID: "24ccc37d-027a-4097-8da5-04df417aef94"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:02:46.915985 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:46.915959    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-5a8f94/premium-w3-5a8f94-kserve-bb96b6848-5hk8h"
Oct 30 21:02:47.052148 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:47.051999    4521 generic.go:356] "Generic (PLEG): container finished" podID="00b6adb9-3733-4ea4-9a26-f3b29991001a" containerID="e361ad15db1741dc7fcce5c213ec24d7f516d1ca35e3e8358688e2c51d1e3813" exitCode=0
Oct 30 21:02:47.086788 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:47.085068    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a8f94/premium-w3-5a8f94-kserve-bb96b6848-5hk8h" podUID="24ccc37d-027a-4097-8da5-04df417aef94" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:02:47.100068 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:47.098481    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-65968b/distinct-2-w5-65968b-kserve-6d65c85f6c-dz5r7" podUID="d6751cf5-7df8-4ffa-b6c8-8b0c4f16ba2a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": dial tcp 10.133.0.93:8000: connect: connection refused"
Oct 30 21:02:47.100068 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:47.099783    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a8f94/simulator-w3-5a8f94-kserve-77cc88cc89-n4sl6" podUID="1fa6900c-eee2-4f00-b0ec-248def48dfc4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": dial tcp 10.133.0.51:8000: connect: connection refused"
Oct 30 21:02:47.106598 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:47.106552    4521 generic.go:356] "Generic (PLEG): container finished" podID="baad1994-acb5-4aae-bb65-a5e8a5b8afdc" containerID="792caf04aadf9b69e35ec7d46269a59947e1107749f2465ea6a82633fae13abd" exitCode=0
Oct 30 21:02:47.174614 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:47.164681    4521 patch_prober.go:29] interesting pod/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-m9wn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:02:47.200690 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:47.200453    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-m9wn4" podUID="9dc1356e-e12d-49f9-822e-f1ccf4fdaa3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:02:47.247348 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:47.229807    4521 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24ccc37d-027a-4097-8da5-04df417aef94-home\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:47.247348 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:47.229901    4521 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24ccc37d-027a-4097-8da5-04df417aef94-tls-certs\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:47.247348 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:47.229916    4521 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24ccc37d-027a-4097-8da5-04df417aef94-model-cache\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:47.247348 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:47.229935    4521 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkrx6\" (UniqueName: \"kubernetes.io/projected/24ccc37d-027a-4097-8da5-04df417aef94-kube-api-access-dkrx6\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:47.247348 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:47.229950    4521 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24ccc37d-027a-4097-8da5-04df417aef94-tmp-dir\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:47.247348 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:47.236373    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24ccc37d-027a-4097-8da5-04df417aef94-dshm" pod "24ccc37d-027a-4097-8da5-04df417aef94" (UID: "24ccc37d-027a-4097-8da5-04df417aef94"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:02:47.332462 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:47.332415    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c2d1eb/premium-w4-c2d1eb-kserve-8696fbd9b9-mxqhm" podUID="4bb0f898-b77f-419d-bff4-6413e82ef51d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:02:47.377957 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:47.345745    4521 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24ccc37d-027a-4097-8da5-04df417aef94-dshm\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:47.411147 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:47.410592    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-5a8f94/premium-w3-5a8f94-kserve-bb96b6848-5hk8h" event={"ID":"24ccc37d-027a-4097-8da5-04df417aef94","Type":"ContainerDied","Data":"198a7db7117f283b4da4bece98ff16b4ab05bdd2e8ca9ed0399ce0186d7067b4"}
Oct 30 21:02:47.411147 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:47.410645    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-tqd8s" event={"ID":"00b6adb9-3733-4ea4-9a26-f3b29991001a","Type":"ContainerDied","Data":"e361ad15db1741dc7fcce5c213ec24d7f516d1ca35e3e8358688e2c51d1e3813"}
Oct 30 21:02:47.411147 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:47.410684    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-p55rm" event={"ID":"baad1994-acb5-4aae-bb65-a5e8a5b8afdc","Type":"ContainerDied","Data":"792caf04aadf9b69e35ec7d46269a59947e1107749f2465ea6a82633fae13abd"}
Oct 30 21:02:47.411147 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:47.410720    4521 scope.go:122] "RemoveContainer" containerID="47c33f2a9ce29670bd78c147b405d8d795a4c4b67ba92d3660b28bff3dd7026c"
Oct 30 21:02:47.443778 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:47.443746    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c2d1eb/premium-w4-c2d1eb-kserve-8696fbd9b9-mxqhm"
Oct 30 21:02:47.456964 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:47.454477    4521 patch_prober.go:29] interesting pod/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-nvwt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:02:47.456964 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:47.454544    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-nvwt8" podUID="6cff2b61-dcbd-4450-b776-43eff57a4fd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:02:47.532398 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:47.532297    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bb0f898-b77f-419d-bff4-6413e82ef51d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4bb0f898-b77f-419d-bff4-6413e82ef51d-model-cache\") pod \"4bb0f898-b77f-419d-bff4-6413e82ef51d\" (UID: \"4bb0f898-b77f-419d-bff4-6413e82ef51d\") "
Oct 30 21:02:47.532774 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:47.532739    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bb0f898-b77f-419d-bff4-6413e82ef51d-kube-api-access-mt6dz\" (UniqueName: \"kubernetes.io/projected/4bb0f898-b77f-419d-bff4-6413e82ef51d-kube-api-access-mt6dz\") pod \"4bb0f898-b77f-419d-bff4-6413e82ef51d\" (UID: \"4bb0f898-b77f-419d-bff4-6413e82ef51d\") "
Oct 30 21:02:47.557342 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:47.555335    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bb0f898-b77f-419d-bff4-6413e82ef51d-model-cache" pod "4bb0f898-b77f-419d-bff4-6413e82ef51d" (UID: "4bb0f898-b77f-419d-bff4-6413e82ef51d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:02:47.568653 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:47.559591    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-65968b/premium-w5-65968b-kserve-67977485f7-c9jpv"
Oct 30 21:02:47.623629 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:47.623588    4521 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4bb0f898-b77f-419d-bff4-6413e82ef51d-model-cache\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:47.776721 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:47.761283    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92b2dd65-8a83-436d-afa8-50e5f0d11fa0-home\" (UniqueName: \"kubernetes.io/empty-dir/92b2dd65-8a83-436d-afa8-50e5f0d11fa0-home\") pod \"92b2dd65-8a83-436d-afa8-50e5f0d11fa0\" (UID: \"92b2dd65-8a83-436d-afa8-50e5f0d11fa0\") "
Oct 30 21:02:47.777100 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:47.777060    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92b2dd65-8a83-436d-afa8-50e5f0d11fa0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92b2dd65-8a83-436d-afa8-50e5f0d11fa0-tmp-dir\") pod \"92b2dd65-8a83-436d-afa8-50e5f0d11fa0\" (UID: \"92b2dd65-8a83-436d-afa8-50e5f0d11fa0\") "
Oct 30 21:02:47.777435 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:47.777411    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92b2dd65-8a83-436d-afa8-50e5f0d11fa0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92b2dd65-8a83-436d-afa8-50e5f0d11fa0-model-cache\") pod \"92b2dd65-8a83-436d-afa8-50e5f0d11fa0\" (UID: \"92b2dd65-8a83-436d-afa8-50e5f0d11fa0\") "
Oct 30 21:02:47.777649 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:47.777630    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bb0f898-b77f-419d-bff4-6413e82ef51d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4bb0f898-b77f-419d-bff4-6413e82ef51d-tmp-dir\") pod \"4bb0f898-b77f-419d-bff4-6413e82ef51d\" (UID: \"4bb0f898-b77f-419d-bff4-6413e82ef51d\") "
Oct 30 21:02:47.777838 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:47.777807    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92b2dd65-8a83-436d-afa8-50e5f0d11fa0-kube-api-access-fw22s\" (UniqueName: \"kubernetes.io/projected/92b2dd65-8a83-436d-afa8-50e5f0d11fa0-kube-api-access-fw22s\") pod \"92b2dd65-8a83-436d-afa8-50e5f0d11fa0\" (UID: \"92b2dd65-8a83-436d-afa8-50e5f0d11fa0\") "
Oct 30 21:02:47.778038 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:47.778009    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4bb0f898-b77f-419d-bff4-6413e82ef51d-tls-certs\" (UniqueName: \"kubernetes.io/secret/4bb0f898-b77f-419d-bff4-6413e82ef51d-tls-certs\") pod \"4bb0f898-b77f-419d-bff4-6413e82ef51d\" (UID: \"4bb0f898-b77f-419d-bff4-6413e82ef51d\") "
Oct 30 21:02:47.778235 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:47.778210    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bb0f898-b77f-419d-bff4-6413e82ef51d-home\" (UniqueName: \"kubernetes.io/empty-dir/4bb0f898-b77f-419d-bff4-6413e82ef51d-home\") pod \"4bb0f898-b77f-419d-bff4-6413e82ef51d\" (UID: \"4bb0f898-b77f-419d-bff4-6413e82ef51d\") "
Oct 30 21:02:47.778394 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:47.778365    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bb0f898-b77f-419d-bff4-6413e82ef51d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4bb0f898-b77f-419d-bff4-6413e82ef51d-dshm\") pod \"4bb0f898-b77f-419d-bff4-6413e82ef51d\" (UID: \"4bb0f898-b77f-419d-bff4-6413e82ef51d\") "
Oct 30 21:02:47.778589 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:47.778571    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/92b2dd65-8a83-436d-afa8-50e5f0d11fa0-tls-certs\" (UniqueName: \"kubernetes.io/secret/92b2dd65-8a83-436d-afa8-50e5f0d11fa0-tls-certs\") pod \"92b2dd65-8a83-436d-afa8-50e5f0d11fa0\" (UID: \"92b2dd65-8a83-436d-afa8-50e5f0d11fa0\") "
Oct 30 21:02:47.778766 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:47.778749    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92b2dd65-8a83-436d-afa8-50e5f0d11fa0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/92b2dd65-8a83-436d-afa8-50e5f0d11fa0-dshm\") pod \"92b2dd65-8a83-436d-afa8-50e5f0d11fa0\" (UID: \"92b2dd65-8a83-436d-afa8-50e5f0d11fa0\") "
Oct 30 21:02:47.787406 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:47.764744    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92b2dd65-8a83-436d-afa8-50e5f0d11fa0-home" pod "92b2dd65-8a83-436d-afa8-50e5f0d11fa0" (UID: "92b2dd65-8a83-436d-afa8-50e5f0d11fa0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:02:47.787406 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:47.783952    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92b2dd65-8a83-436d-afa8-50e5f0d11fa0-tmp-dir" pod "92b2dd65-8a83-436d-afa8-50e5f0d11fa0" (UID: "92b2dd65-8a83-436d-afa8-50e5f0d11fa0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:02:47.787406 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:47.784131    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92b2dd65-8a83-436d-afa8-50e5f0d11fa0-model-cache" pod "92b2dd65-8a83-436d-afa8-50e5f0d11fa0" (UID: "92b2dd65-8a83-436d-afa8-50e5f0d11fa0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:02:47.787406 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:47.784294    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bb0f898-b77f-419d-bff4-6413e82ef51d-tmp-dir" pod "4bb0f898-b77f-419d-bff4-6413e82ef51d" (UID: "4bb0f898-b77f-419d-bff4-6413e82ef51d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:02:47.802070 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:47.802029    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bb0f898-b77f-419d-bff4-6413e82ef51d-home" pod "4bb0f898-b77f-419d-bff4-6413e82ef51d" (UID: "4bb0f898-b77f-419d-bff4-6413e82ef51d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:02:47.877508 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:47.877464    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4bb0f898-b77f-419d-bff4-6413e82ef51d-tls-certs" pod "4bb0f898-b77f-419d-bff4-6413e82ef51d" (UID: "4bb0f898-b77f-419d-bff4-6413e82ef51d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:02:47.895203 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:47.886463    4521 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4bb0f898-b77f-419d-bff4-6413e82ef51d-tls-certs\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:47.896855 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:47.895058    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bb0f898-b77f-419d-bff4-6413e82ef51d-kube-api-access-mt6dz" pod "4bb0f898-b77f-419d-bff4-6413e82ef51d" (UID: "4bb0f898-b77f-419d-bff4-6413e82ef51d"). InnerVolumeSpecName "kube-api-access-mt6dz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:02:47.897015 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:47.895369    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92b2dd65-8a83-436d-afa8-50e5f0d11fa0-kube-api-access-fw22s" pod "92b2dd65-8a83-436d-afa8-50e5f0d11fa0" (UID: "92b2dd65-8a83-436d-afa8-50e5f0d11fa0"). InnerVolumeSpecName "kube-api-access-fw22s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:02:47.897113 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:47.895466    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92b2dd65-8a83-436d-afa8-50e5f0d11fa0-dshm" pod "92b2dd65-8a83-436d-afa8-50e5f0d11fa0" (UID: "92b2dd65-8a83-436d-afa8-50e5f0d11fa0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:02:47.897231 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:47.895542    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bb0f898-b77f-419d-bff4-6413e82ef51d-dshm" pod "4bb0f898-b77f-419d-bff4-6413e82ef51d" (UID: "4bb0f898-b77f-419d-bff4-6413e82ef51d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:02:47.897333 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:47.895538    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92b2dd65-8a83-436d-afa8-50e5f0d11fa0-tls-certs" pod "92b2dd65-8a83-436d-afa8-50e5f0d11fa0" (UID: "92b2dd65-8a83-436d-afa8-50e5f0d11fa0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:02:47.897468 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:47.895765    4521 patch_prober.go:29] interesting pod/e2e-worker-w5-65968b-openshift-default-68896fcddc-sbpfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:02:47.898471 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:47.897603    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-sbpfk" podUID="7a5f91bb-7229-454f-a006-0d768f8879f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:02:47.898696 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:47.897821    4521 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4bb0f898-b77f-419d-bff4-6413e82ef51d-home\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:47.898852 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:47.898838    4521 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92b2dd65-8a83-436d-afa8-50e5f0d11fa0-home\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:47.898970 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:47.898957    4521 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92b2dd65-8a83-436d-afa8-50e5f0d11fa0-tmp-dir\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:47.899092 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:47.899077    4521 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92b2dd65-8a83-436d-afa8-50e5f0d11fa0-model-cache\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:47.899210 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:47.899196    4521 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4bb0f898-b77f-419d-bff4-6413e82ef51d-tmp-dir\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:48.000954 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.000613    4521 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mt6dz\" (UniqueName: \"kubernetes.io/projected/4bb0f898-b77f-419d-bff4-6413e82ef51d-kube-api-access-mt6dz\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:48.000954 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.000656    4521 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fw22s\" (UniqueName: \"kubernetes.io/projected/92b2dd65-8a83-436d-afa8-50e5f0d11fa0-kube-api-access-fw22s\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:48.000954 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.000672    4521 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4bb0f898-b77f-419d-bff4-6413e82ef51d-dshm\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:48.000954 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.000692    4521 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92b2dd65-8a83-436d-afa8-50e5f0d11fa0-tls-certs\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:48.000954 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.000712    4521 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92b2dd65-8a83-436d-afa8-50e5f0d11fa0-dshm\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:48.014947 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.014897    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-65968b/premium-w5-65968b-kserve-67977485f7-c9jpv" podUID="92b2dd65-8a83-436d-afa8-50e5f0d11fa0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:02:48.114350 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.090702    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-p55rm"
Oct 30 21:02:48.115977 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.115941    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/baad1994-acb5-4aae-bb65-a5e8a5b8afdc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/baad1994-acb5-4aae-bb65-a5e8a5b8afdc-istio-envoy\") pod \"baad1994-acb5-4aae-bb65-a5e8a5b8afdc\" (UID: \"baad1994-acb5-4aae-bb65-a5e8a5b8afdc\") "
Oct 30 21:02:48.126271 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.125598    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/baad1994-acb5-4aae-bb65-a5e8a5b8afdc-istio-token\" (UniqueName: \"kubernetes.io/projected/baad1994-acb5-4aae-bb65-a5e8a5b8afdc-istio-token\") pod \"baad1994-acb5-4aae-bb65-a5e8a5b8afdc\" (UID: \"baad1994-acb5-4aae-bb65-a5e8a5b8afdc\") "
Oct 30 21:02:48.126271 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.125677    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/baad1994-acb5-4aae-bb65-a5e8a5b8afdc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/baad1994-acb5-4aae-bb65-a5e8a5b8afdc-workload-certs\") pod \"baad1994-acb5-4aae-bb65-a5e8a5b8afdc\" (UID: \"baad1994-acb5-4aae-bb65-a5e8a5b8afdc\") "
Oct 30 21:02:48.126271 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.125786    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/baad1994-acb5-4aae-bb65-a5e8a5b8afdc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/baad1994-acb5-4aae-bb65-a5e8a5b8afdc-workload-socket\") pod \"baad1994-acb5-4aae-bb65-a5e8a5b8afdc\" (UID: \"baad1994-acb5-4aae-bb65-a5e8a5b8afdc\") "
Oct 30 21:02:48.126271 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.125852    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/baad1994-acb5-4aae-bb65-a5e8a5b8afdc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/baad1994-acb5-4aae-bb65-a5e8a5b8afdc-istio-podinfo\") pod \"baad1994-acb5-4aae-bb65-a5e8a5b8afdc\" (UID: \"baad1994-acb5-4aae-bb65-a5e8a5b8afdc\") "
Oct 30 21:02:48.126271 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.125903    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/baad1994-acb5-4aae-bb65-a5e8a5b8afdc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/baad1994-acb5-4aae-bb65-a5e8a5b8afdc-credential-socket\") pod \"baad1994-acb5-4aae-bb65-a5e8a5b8afdc\" (UID: \"baad1994-acb5-4aae-bb65-a5e8a5b8afdc\") "
Oct 30 21:02:48.126271 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.125972    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/baad1994-acb5-4aae-bb65-a5e8a5b8afdc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/baad1994-acb5-4aae-bb65-a5e8a5b8afdc-istiod-ca-cert\") pod \"baad1994-acb5-4aae-bb65-a5e8a5b8afdc\" (UID: \"baad1994-acb5-4aae-bb65-a5e8a5b8afdc\") "
Oct 30 21:02:48.126271 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.126016    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/baad1994-acb5-4aae-bb65-a5e8a5b8afdc-kube-api-access-qtwp7\" (UniqueName: \"kubernetes.io/projected/baad1994-acb5-4aae-bb65-a5e8a5b8afdc-kube-api-access-qtwp7\") pod \"baad1994-acb5-4aae-bb65-a5e8a5b8afdc\" (UID: \"baad1994-acb5-4aae-bb65-a5e8a5b8afdc\") "
Oct 30 21:02:48.126271 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.126070    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/baad1994-acb5-4aae-bb65-a5e8a5b8afdc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/baad1994-acb5-4aae-bb65-a5e8a5b8afdc-istio-data\") pod \"baad1994-acb5-4aae-bb65-a5e8a5b8afdc\" (UID: \"baad1994-acb5-4aae-bb65-a5e8a5b8afdc\") "
Oct 30 21:02:48.139043 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.119937    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-5a8f94/simulator-w3-5a8f94-kserve-77cc88cc89-n4sl6"
Oct 30 21:02:48.139480 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.139399    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baad1994-acb5-4aae-bb65-a5e8a5b8afdc-credential-socket" pod "baad1994-acb5-4aae-bb65-a5e8a5b8afdc" (UID: "baad1994-acb5-4aae-bb65-a5e8a5b8afdc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:02:48.141553 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.139842    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baad1994-acb5-4aae-bb65-a5e8a5b8afdc-workload-certs" pod "baad1994-acb5-4aae-bb65-a5e8a5b8afdc" (UID: "baad1994-acb5-4aae-bb65-a5e8a5b8afdc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:02:48.141553 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.140545    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-tqd8s"
Oct 30 21:02:48.150542 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.149483    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baad1994-acb5-4aae-bb65-a5e8a5b8afdc-workload-socket" pod "baad1994-acb5-4aae-bb65-a5e8a5b8afdc" (UID: "baad1994-acb5-4aae-bb65-a5e8a5b8afdc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:02:48.150542 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.149720    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baad1994-acb5-4aae-bb65-a5e8a5b8afdc-istio-data" pod "baad1994-acb5-4aae-bb65-a5e8a5b8afdc" (UID: "baad1994-acb5-4aae-bb65-a5e8a5b8afdc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:02:48.181401 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.159868    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/baad1994-acb5-4aae-bb65-a5e8a5b8afdc-istiod-ca-cert" pod "baad1994-acb5-4aae-bb65-a5e8a5b8afdc" (UID: "baad1994-acb5-4aae-bb65-a5e8a5b8afdc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:02:48.203045 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.202463    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/baad1994-acb5-4aae-bb65-a5e8a5b8afdc-istio-podinfo" pod "baad1994-acb5-4aae-bb65-a5e8a5b8afdc" (UID: "baad1994-acb5-4aae-bb65-a5e8a5b8afdc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:02:48.221600 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.205204    4521 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-5a8f94/premium-w3-5a8f94-kserve-bb96b6848-5hk8h"]
Oct 30 21:02:48.236462 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.228007    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fa6900c-eee2-4f00-b0ec-248def48dfc4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1fa6900c-eee2-4f00-b0ec-248def48dfc4-model-cache\") pod \"1fa6900c-eee2-4f00-b0ec-248def48dfc4\" (UID: \"1fa6900c-eee2-4f00-b0ec-248def48dfc4\") "
Oct 30 21:02:48.236462 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.228142    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fa6900c-eee2-4f00-b0ec-248def48dfc4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1fa6900c-eee2-4f00-b0ec-248def48dfc4-dshm\") pod \"1fa6900c-eee2-4f00-b0ec-248def48dfc4\" (UID: \"1fa6900c-eee2-4f00-b0ec-248def48dfc4\") "
Oct 30 21:02:48.236462 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.228208    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fa6900c-eee2-4f00-b0ec-248def48dfc4-kube-api-access-xljl7\" (UniqueName: \"kubernetes.io/projected/1fa6900c-eee2-4f00-b0ec-248def48dfc4-kube-api-access-xljl7\") pod \"1fa6900c-eee2-4f00-b0ec-248def48dfc4\" (UID: \"1fa6900c-eee2-4f00-b0ec-248def48dfc4\") "
Oct 30 21:02:48.236462 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.228296    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00b6adb9-3733-4ea4-9a26-f3b29991001a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00b6adb9-3733-4ea4-9a26-f3b29991001a-istiod-ca-cert\") pod \"00b6adb9-3733-4ea4-9a26-f3b29991001a\" (UID: \"00b6adb9-3733-4ea4-9a26-f3b29991001a\") "
Oct 30 21:02:48.236462 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.228347    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fa6900c-eee2-4f00-b0ec-248def48dfc4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fa6900c-eee2-4f00-b0ec-248def48dfc4-tmp-dir\") pod \"1fa6900c-eee2-4f00-b0ec-248def48dfc4\" (UID: \"1fa6900c-eee2-4f00-b0ec-248def48dfc4\") "
Oct 30 21:02:48.236462 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.228472    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00b6adb9-3733-4ea4-9a26-f3b29991001a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00b6adb9-3733-4ea4-9a26-f3b29991001a-workload-certs\") pod \"00b6adb9-3733-4ea4-9a26-f3b29991001a\" (UID: \"00b6adb9-3733-4ea4-9a26-f3b29991001a\") "
Oct 30 21:02:48.236462 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.228512    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fa6900c-eee2-4f00-b0ec-248def48dfc4-home\" (UniqueName: \"kubernetes.io/empty-dir/1fa6900c-eee2-4f00-b0ec-248def48dfc4-home\") pod \"1fa6900c-eee2-4f00-b0ec-248def48dfc4\" (UID: \"1fa6900c-eee2-4f00-b0ec-248def48dfc4\") "
Oct 30 21:02:48.236462 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.228564    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00b6adb9-3733-4ea4-9a26-f3b29991001a-istio-token\" (UniqueName: \"kubernetes.io/projected/00b6adb9-3733-4ea4-9a26-f3b29991001a-istio-token\") pod \"00b6adb9-3733-4ea4-9a26-f3b29991001a\" (UID: \"00b6adb9-3733-4ea4-9a26-f3b29991001a\") "
Oct 30 21:02:48.236462 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.228595    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00b6adb9-3733-4ea4-9a26-f3b29991001a-kube-api-access-xmrvp\" (UniqueName: \"kubernetes.io/projected/00b6adb9-3733-4ea4-9a26-f3b29991001a-kube-api-access-xmrvp\") pod \"00b6adb9-3733-4ea4-9a26-f3b29991001a\" (UID: \"00b6adb9-3733-4ea4-9a26-f3b29991001a\") "
Oct 30 21:02:48.236462 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.228681    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/00b6adb9-3733-4ea4-9a26-f3b29991001a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00b6adb9-3733-4ea4-9a26-f3b29991001a-istio-podinfo\") pod \"00b6adb9-3733-4ea4-9a26-f3b29991001a\" (UID: \"00b6adb9-3733-4ea4-9a26-f3b29991001a\") "
Oct 30 21:02:48.236462 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.228716    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00b6adb9-3733-4ea4-9a26-f3b29991001a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00b6adb9-3733-4ea4-9a26-f3b29991001a-credential-socket\") pod \"00b6adb9-3733-4ea4-9a26-f3b29991001a\" (UID: \"00b6adb9-3733-4ea4-9a26-f3b29991001a\") "
Oct 30 21:02:48.236462 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.228758    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1fa6900c-eee2-4f00-b0ec-248def48dfc4-tls-certs\" (UniqueName: \"kubernetes.io/secret/1fa6900c-eee2-4f00-b0ec-248def48dfc4-tls-certs\") pod \"1fa6900c-eee2-4f00-b0ec-248def48dfc4\" (UID: \"1fa6900c-eee2-4f00-b0ec-248def48dfc4\") "
Oct 30 21:02:48.236462 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.228797    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00b6adb9-3733-4ea4-9a26-f3b29991001a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00b6adb9-3733-4ea4-9a26-f3b29991001a-istio-data\") pod \"00b6adb9-3733-4ea4-9a26-f3b29991001a\" (UID: \"00b6adb9-3733-4ea4-9a26-f3b29991001a\") "
Oct 30 21:02:48.236462 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.228842    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00b6adb9-3733-4ea4-9a26-f3b29991001a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00b6adb9-3733-4ea4-9a26-f3b29991001a-istio-envoy\") pod \"00b6adb9-3733-4ea4-9a26-f3b29991001a\" (UID: \"00b6adb9-3733-4ea4-9a26-f3b29991001a\") "
Oct 30 21:02:48.236462 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.228878    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00b6adb9-3733-4ea4-9a26-f3b29991001a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00b6adb9-3733-4ea4-9a26-f3b29991001a-workload-socket\") pod \"00b6adb9-3733-4ea4-9a26-f3b29991001a\" (UID: \"00b6adb9-3733-4ea4-9a26-f3b29991001a\") "
Oct 30 21:02:48.236462 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.229568    4521 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/baad1994-acb5-4aae-bb65-a5e8a5b8afdc-workload-certs\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:48.236462 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.229592    4521 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/baad1994-acb5-4aae-bb65-a5e8a5b8afdc-workload-socket\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:48.236462 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.229610    4521 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/baad1994-acb5-4aae-bb65-a5e8a5b8afdc-istio-podinfo\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:48.236462 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.229626    4521 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/baad1994-acb5-4aae-bb65-a5e8a5b8afdc-credential-socket\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:48.236462 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.229641    4521 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/baad1994-acb5-4aae-bb65-a5e8a5b8afdc-istiod-ca-cert\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:48.236462 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.229655    4521 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/baad1994-acb5-4aae-bb65-a5e8a5b8afdc-istio-data\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:48.236462 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.235435    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baad1994-acb5-4aae-bb65-a5e8a5b8afdc-istio-envoy" pod "baad1994-acb5-4aae-bb65-a5e8a5b8afdc" (UID: "baad1994-acb5-4aae-bb65-a5e8a5b8afdc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:02:48.236462 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.235493    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fa6900c-eee2-4f00-b0ec-248def48dfc4-home" pod "1fa6900c-eee2-4f00-b0ec-248def48dfc4" (UID: "1fa6900c-eee2-4f00-b0ec-248def48dfc4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:02:48.236462 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.235729    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fa6900c-eee2-4f00-b0ec-248def48dfc4-model-cache" pod "1fa6900c-eee2-4f00-b0ec-248def48dfc4" (UID: "1fa6900c-eee2-4f00-b0ec-248def48dfc4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:02:48.258857 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.257411    4521 patch_prober.go:29] interesting pod/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-m9wn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:02:48.258857 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.257473    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-m9wn4" podUID="9dc1356e-e12d-49f9-822e-f1ccf4fdaa3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:02:48.258857 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.258430    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00b6adb9-3733-4ea4-9a26-f3b29991001a-istio-data" pod "00b6adb9-3733-4ea4-9a26-f3b29991001a" (UID: "00b6adb9-3733-4ea4-9a26-f3b29991001a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:02:48.258857 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.258485    4521 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-5a8f94/premium-w3-5a8f94-kserve-bb96b6848-5hk8h"]
Oct 30 21:02:48.260162 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.260134    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00b6adb9-3733-4ea4-9a26-f3b29991001a-credential-socket" pod "00b6adb9-3733-4ea4-9a26-f3b29991001a" (UID: "00b6adb9-3733-4ea4-9a26-f3b29991001a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:02:48.268093 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.263484    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a8f94/distinct-w3-5a8f94-kserve-5f8b544d59-mf95j" podUID="b8ac1cca-4ebd-474c-b32f-a3595a4cd847" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:02:48.268589 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.268561    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/baad1994-acb5-4aae-bb65-a5e8a5b8afdc-kube-api-access-qtwp7" pod "baad1994-acb5-4aae-bb65-a5e8a5b8afdc" (UID: "baad1994-acb5-4aae-bb65-a5e8a5b8afdc"). InnerVolumeSpecName "kube-api-access-qtwp7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:02:48.281947 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.281712    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fa6900c-eee2-4f00-b0ec-248def48dfc4-tmp-dir" pod "1fa6900c-eee2-4f00-b0ec-248def48dfc4" (UID: "1fa6900c-eee2-4f00-b0ec-248def48dfc4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:02:48.288350 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.282239    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00b6adb9-3733-4ea4-9a26-f3b29991001a-istiod-ca-cert" pod "00b6adb9-3733-4ea4-9a26-f3b29991001a" (UID: "00b6adb9-3733-4ea4-9a26-f3b29991001a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:02:48.288692 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.288591    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00b6adb9-3733-4ea4-9a26-f3b29991001a-workload-certs" pod "00b6adb9-3733-4ea4-9a26-f3b29991001a" (UID: "00b6adb9-3733-4ea4-9a26-f3b29991001a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:02:48.289677 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.289649    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/baad1994-acb5-4aae-bb65-a5e8a5b8afdc-istio-token" pod "baad1994-acb5-4aae-bb65-a5e8a5b8afdc" (UID: "baad1994-acb5-4aae-bb65-a5e8a5b8afdc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:02:48.294718 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.294491    4521 generic.go:356] "Generic (PLEG): container finished" podID="f6f47707-ed2c-4860-a13a-b1a5d9d1bbe3" containerID="8b8bd567c67d21ef122a8fd6141d1c042d7bfca248ef665aa84d0e993110b661" exitCode=0
Oct 30 21:02:48.294718 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.294623    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-hmwtj" event={"ID":"f6f47707-ed2c-4860-a13a-b1a5d9d1bbe3","Type":"ContainerDied","Data":"8b8bd567c67d21ef122a8fd6141d1c042d7bfca248ef665aa84d0e993110b661"}
Oct 30 21:02:48.297913 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.297885    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-408918eb-f26js" event={"ID":"7d10d07e-a60e-4c6a-b067-66234939fa57","Type":"ContainerStarted","Data":"5cf7f7a8b1f335911c58c99ac404c84165377fe9f2d6e87979e4ec7dc0d1b324"}
Oct 30 21:02:48.302544 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.302417    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00b6adb9-3733-4ea4-9a26-f3b29991001a-workload-socket" pod "00b6adb9-3733-4ea4-9a26-f3b29991001a" (UID: "00b6adb9-3733-4ea4-9a26-f3b29991001a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:02:48.305349 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.303394    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c2d1eb/premium-w4-c2d1eb-kserve-8696fbd9b9-mxqhm" event={"ID":"4bb0f898-b77f-419d-bff4-6413e82ef51d","Type":"ContainerDied","Data":"6e0b81cd2e63852db6b987dae98cfeaddecd229881b780ebd6f64910abbccaab"}
Oct 30 21:02:48.305349 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.303440    4521 scope.go:122] "RemoveContainer" containerID="cf06834ad798b61733cac66d7babac65e2e8816c2cc87cdfd0eba97e9cd9d770"
Oct 30 21:02:48.305349 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.303542    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c2d1eb/premium-w4-c2d1eb-kserve-8696fbd9b9-mxqhm"
Oct 30 21:02:48.329507 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.316370    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-5a8f94/distinct-w3-5a8f94-kserve-5f8b544d59-mf95j" event={"ID":"b8ac1cca-4ebd-474c-b32f-a3595a4cd847","Type":"ContainerDied","Data":"b7e094cf578f7e7de41fa75caeab6a454f555740d869f20455f6b1c795712b83"}
Oct 30 21:02:48.329507 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.316446    4521 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b7e094cf578f7e7de41fa75caeab6a454f555740d869f20455f6b1c795712b83"
Oct 30 21:02:48.339649 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.339605    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c2d1eb/premium-w4-c2d1eb-kserve-8696fbd9b9-mxqhm" podUID="4bb0f898-b77f-419d-bff4-6413e82ef51d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded"
Oct 30 21:02:48.342157 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.342129    4521 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fa6900c-eee2-4f00-b0ec-248def48dfc4-tmp-dir\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:48.342353 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.342315    4521 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00b6adb9-3733-4ea4-9a26-f3b29991001a-workload-certs\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:48.342465 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.342451    4521 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1fa6900c-eee2-4f00-b0ec-248def48dfc4-home\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:48.342578 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.342565    4521 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00b6adb9-3733-4ea4-9a26-f3b29991001a-credential-socket\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:48.342679 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.342666    4521 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00b6adb9-3733-4ea4-9a26-f3b29991001a-istio-data\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:48.342780 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.342767    4521 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtwp7\" (UniqueName: \"kubernetes.io/projected/baad1994-acb5-4aae-bb65-a5e8a5b8afdc-kube-api-access-qtwp7\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:48.342874 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.342859    4521 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00b6adb9-3733-4ea4-9a26-f3b29991001a-workload-socket\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:48.342962 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.342950    4521 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1fa6900c-eee2-4f00-b0ec-248def48dfc4-model-cache\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:48.343060 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.343046    4521 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/baad1994-acb5-4aae-bb65-a5e8a5b8afdc-istio-envoy\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:48.343165 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.343153    4521 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/baad1994-acb5-4aae-bb65-a5e8a5b8afdc-istio-token\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:48.343252 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.343240    4521 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00b6adb9-3733-4ea4-9a26-f3b29991001a-istiod-ca-cert\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:48.345782 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.345476    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-tqd8s" event={"ID":"00b6adb9-3733-4ea4-9a26-f3b29991001a","Type":"ContainerDied","Data":"bb33ed467995ec4b4dc448e740cf95ca7cc41a8880caa5ddc13f3763b923e2a0"}
Oct 30 21:02:48.345782 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.345597    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-tqd8s"
Oct 30 21:02:48.348048 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.347800    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-5a8f94/simulator-w3-5a8f94-kserve-77cc88cc89-n4sl6" event={"ID":"1fa6900c-eee2-4f00-b0ec-248def48dfc4","Type":"ContainerDied","Data":"511ae6fb670346702d3657af83f03c55f096748f636134b00370d78cff90ee85"}
Oct 30 21:02:48.348048 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.347890    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-5a8f94/simulator-w3-5a8f94-kserve-77cc88cc89-n4sl6"
Oct 30 21:02:48.350057 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.350033    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-65968b/premium-w5-65968b-kserve-67977485f7-c9jpv" event={"ID":"92b2dd65-8a83-436d-afa8-50e5f0d11fa0","Type":"ContainerDied","Data":"0a1316c5e82c70ca87620739b7f948403e37a79683bedce33aa4815201fdffc1"}
Oct 30 21:02:48.350245 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.350228    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-65968b/premium-w5-65968b-kserve-67977485f7-c9jpv"
Oct 30 21:02:48.357048 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.356630    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-65968b/distinct-2-w5-65968b-kserve-6d65c85f6c-dz5r7" event={"ID":"d6751cf5-7df8-4ffa-b6c8-8b0c4f16ba2a","Type":"ContainerDied","Data":"ddcc10eae2773392a1659a2c8896257485ab56351be3d2d1e78fe797306209f0"}
Oct 30 21:02:48.357048 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.356679    4521 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ddcc10eae2773392a1659a2c8896257485ab56351be3d2d1e78fe797306209f0"
Oct 30 21:02:48.358696 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.358626    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-5a8f94/distinct-2-w3-5a8f94-kserve-ffcc6449b-hv48k" event={"ID":"bc0f5352-5c44-4c35-adf5-3fffab06b644","Type":"ContainerDied","Data":"de7e2b95cd5b666bab8c0c5939d56b6fe2ee541f8412a447342eff576b5821e5"}
Oct 30 21:02:48.358696 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.358660    4521 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="de7e2b95cd5b666bab8c0c5939d56b6fe2ee541f8412a447342eff576b5821e5"
Oct 30 21:02:48.368696 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.367693    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-p55rm" event={"ID":"baad1994-acb5-4aae-bb65-a5e8a5b8afdc","Type":"ContainerDied","Data":"150b3e052e1edf9d3da1f6bec81a4b61348e9462093242ccf0b16d35cefd0589"}
Oct 30 21:02:48.368696 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.367805    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-p55rm"
Oct 30 21:02:48.369431 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.369209    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-5a8f94/distinct-2-w3-5a8f94-kserve-ffcc6449b-hv48k"
Oct 30 21:02:48.369683 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.369666    4521 scope.go:122] "RemoveContainer" containerID="e361ad15db1741dc7fcce5c213ec24d7f516d1ca35e3e8358688e2c51d1e3813"
Oct 30 21:02:48.373872 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.373439    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00b6adb9-3733-4ea4-9a26-f3b29991001a-istio-token" pod "00b6adb9-3733-4ea4-9a26-f3b29991001a" (UID: "00b6adb9-3733-4ea4-9a26-f3b29991001a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:02:48.374301 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.374265    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fa6900c-eee2-4f00-b0ec-248def48dfc4-dshm" pod "1fa6900c-eee2-4f00-b0ec-248def48dfc4" (UID: "1fa6900c-eee2-4f00-b0ec-248def48dfc4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:02:48.387963 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.379119    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00b6adb9-3733-4ea4-9a26-f3b29991001a-istio-envoy" pod "00b6adb9-3733-4ea4-9a26-f3b29991001a" (UID: "00b6adb9-3733-4ea4-9a26-f3b29991001a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:02:48.387963 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.379254    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fa6900c-eee2-4f00-b0ec-248def48dfc4-kube-api-access-xljl7" pod "1fa6900c-eee2-4f00-b0ec-248def48dfc4" (UID: "1fa6900c-eee2-4f00-b0ec-248def48dfc4"). InnerVolumeSpecName "kube-api-access-xljl7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:02:48.387963 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.379386    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1fa6900c-eee2-4f00-b0ec-248def48dfc4-tls-certs" pod "1fa6900c-eee2-4f00-b0ec-248def48dfc4" (UID: "1fa6900c-eee2-4f00-b0ec-248def48dfc4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:02:48.387963 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.380506    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-5a8f94/distinct-w3-5a8f94-kserve-5f8b544d59-mf95j"
Oct 30 21:02:48.406815 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.406787    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-65968b/distinct-2-w5-65968b-kserve-6d65c85f6c-dz5r7"
Oct 30 21:02:48.433658 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.433617    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/00b6adb9-3733-4ea4-9a26-f3b29991001a-istio-podinfo" pod "00b6adb9-3733-4ea4-9a26-f3b29991001a" (UID: "00b6adb9-3733-4ea4-9a26-f3b29991001a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:02:48.434518 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.434471    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00b6adb9-3733-4ea4-9a26-f3b29991001a-kube-api-access-xmrvp" pod "00b6adb9-3733-4ea4-9a26-f3b29991001a" (UID: "00b6adb9-3733-4ea4-9a26-f3b29991001a"). InnerVolumeSpecName "kube-api-access-xmrvp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:02:48.434642 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.434594    4521 scope.go:122] "RemoveContainer" containerID="155ef94dbc2d28a469b16f4b47f77b5b424a93c6c7b687308f435f40ee44d4cc"
Oct 30 21:02:48.465567 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.449819    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8ac1cca-4ebd-474c-b32f-a3595a4cd847-kube-api-access-bstjz\" (UniqueName: \"kubernetes.io/projected/b8ac1cca-4ebd-474c-b32f-a3595a4cd847-kube-api-access-bstjz\") pod \"b8ac1cca-4ebd-474c-b32f-a3595a4cd847\" (UID: \"b8ac1cca-4ebd-474c-b32f-a3595a4cd847\") "
Oct 30 21:02:48.465567 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.449886    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc0f5352-5c44-4c35-adf5-3fffab06b644-kube-api-access-tshfg\" (UniqueName: \"kubernetes.io/projected/bc0f5352-5c44-4c35-adf5-3fffab06b644-kube-api-access-tshfg\") pod \"bc0f5352-5c44-4c35-adf5-3fffab06b644\" (UID: \"bc0f5352-5c44-4c35-adf5-3fffab06b644\") "
Oct 30 21:02:48.465567 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.449928    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8ac1cca-4ebd-474c-b32f-a3595a4cd847-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8ac1cca-4ebd-474c-b32f-a3595a4cd847-tmp-dir\") pod \"b8ac1cca-4ebd-474c-b32f-a3595a4cd847\" (UID: \"b8ac1cca-4ebd-474c-b32f-a3595a4cd847\") "
Oct 30 21:02:48.465567 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.449966    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc0f5352-5c44-4c35-adf5-3fffab06b644-dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc0f5352-5c44-4c35-adf5-3fffab06b644-dshm\") pod \"bc0f5352-5c44-4c35-adf5-3fffab06b644\" (UID: \"bc0f5352-5c44-4c35-adf5-3fffab06b644\") "
Oct 30 21:02:48.465567 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.450026    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc0f5352-5c44-4c35-adf5-3fffab06b644-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc0f5352-5c44-4c35-adf5-3fffab06b644-tmp-dir\") pod \"bc0f5352-5c44-4c35-adf5-3fffab06b644\" (UID: \"bc0f5352-5c44-4c35-adf5-3fffab06b644\") "
Oct 30 21:02:48.465567 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.450075    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8ac1cca-4ebd-474c-b32f-a3595a4cd847-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8ac1cca-4ebd-474c-b32f-a3595a4cd847-model-cache\") pod \"b8ac1cca-4ebd-474c-b32f-a3595a4cd847\" (UID: \"b8ac1cca-4ebd-474c-b32f-a3595a4cd847\") "
Oct 30 21:02:48.533398 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.470502    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8ac1cca-4ebd-474c-b32f-a3595a4cd847-tmp-dir" pod "b8ac1cca-4ebd-474c-b32f-a3595a4cd847" (UID: "b8ac1cca-4ebd-474c-b32f-a3595a4cd847"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:02:48.533398 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.477638    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc0f5352-5c44-4c35-adf5-3fffab06b644-tmp-dir" pod "bc0f5352-5c44-4c35-adf5-3fffab06b644" (UID: "bc0f5352-5c44-4c35-adf5-3fffab06b644"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:02:48.533398 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.478472    4521 patch_prober.go:29] interesting pod/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-nvwt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:02:48.533398 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.478523    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-nvwt8" podUID="6cff2b61-dcbd-4450-b776-43eff57a4fd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:02:48.533398 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.478570    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc0f5352-5c44-4c35-adf5-3fffab06b644-kube-api-access-tshfg" pod "bc0f5352-5c44-4c35-adf5-3fffab06b644" (UID: "bc0f5352-5c44-4c35-adf5-3fffab06b644"). InnerVolumeSpecName "kube-api-access-tshfg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:02:48.533398 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.491196    4521 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-408918eb-f26js" podStartSLOduration=8.49117313 podStartE2EDuration="8.49117313s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:02:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:02:48.484109235 +0000 UTC m=+1435.829730486" watchObservedRunningTime="2026-10-30 21:02:48.49117313 +0000 UTC m=+1435.836794370"
Oct 30 21:02:48.533398 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.492054    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8ac1cca-4ebd-474c-b32f-a3595a4cd847-model-cache" pod "b8ac1cca-4ebd-474c-b32f-a3595a4cd847" (UID: "b8ac1cca-4ebd-474c-b32f-a3595a4cd847"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:02:48.533398 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.499128    4521 scope.go:122] "RemoveContainer" containerID="f850ab3fba071258f1ebd3015d73e51d8c7a3c2027a8c8a5c70910ca96df1a37"
Oct 30 21:02:48.533398 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.532559    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc0f5352-5c44-4c35-adf5-3fffab06b644-tls-certs\" (UniqueName: \"kubernetes.io/secret/bc0f5352-5c44-4c35-adf5-3fffab06b644-tls-certs\") pod \"bc0f5352-5c44-4c35-adf5-3fffab06b644\" (UID: \"bc0f5352-5c44-4c35-adf5-3fffab06b644\") "
Oct 30 21:02:48.533398 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.532612    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6751cf5-7df8-4ffa-b6c8-8b0c4f16ba2a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6751cf5-7df8-4ffa-b6c8-8b0c4f16ba2a-tmp-dir\") pod \"d6751cf5-7df8-4ffa-b6c8-8b0c4f16ba2a\" (UID: \"d6751cf5-7df8-4ffa-b6c8-8b0c4f16ba2a\") "
Oct 30 21:02:48.533398 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.532712    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8ac1cca-4ebd-474c-b32f-a3595a4cd847-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8ac1cca-4ebd-474c-b32f-a3595a4cd847-dshm\") pod \"b8ac1cca-4ebd-474c-b32f-a3595a4cd847\" (UID: \"b8ac1cca-4ebd-474c-b32f-a3595a4cd847\") "
Oct 30 21:02:48.533398 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.532772    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6751cf5-7df8-4ffa-b6c8-8b0c4f16ba2a-home\" (UniqueName: \"kubernetes.io/empty-dir/d6751cf5-7df8-4ffa-b6c8-8b0c4f16ba2a-home\") pod \"d6751cf5-7df8-4ffa-b6c8-8b0c4f16ba2a\" (UID: \"d6751cf5-7df8-4ffa-b6c8-8b0c4f16ba2a\") "
Oct 30 21:02:48.533398 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.532800    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc0f5352-5c44-4c35-adf5-3fffab06b644-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc0f5352-5c44-4c35-adf5-3fffab06b644-model-cache\") pod \"bc0f5352-5c44-4c35-adf5-3fffab06b644\" (UID: \"bc0f5352-5c44-4c35-adf5-3fffab06b644\") "
Oct 30 21:02:48.533398 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.532865    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b8ac1cca-4ebd-474c-b32f-a3595a4cd847-tls-certs\" (UniqueName: \"kubernetes.io/secret/b8ac1cca-4ebd-474c-b32f-a3595a4cd847-tls-certs\") pod \"b8ac1cca-4ebd-474c-b32f-a3595a4cd847\" (UID: \"b8ac1cca-4ebd-474c-b32f-a3595a4cd847\") "
Oct 30 21:02:48.533398 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.532922    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6751cf5-7df8-4ffa-b6c8-8b0c4f16ba2a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6751cf5-7df8-4ffa-b6c8-8b0c4f16ba2a-model-cache\") pod \"d6751cf5-7df8-4ffa-b6c8-8b0c4f16ba2a\" (UID: \"d6751cf5-7df8-4ffa-b6c8-8b0c4f16ba2a\") "
Oct 30 21:02:48.533398 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.532991    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8ac1cca-4ebd-474c-b32f-a3595a4cd847-home\" (UniqueName: \"kubernetes.io/empty-dir/b8ac1cca-4ebd-474c-b32f-a3595a4cd847-home\") pod \"b8ac1cca-4ebd-474c-b32f-a3595a4cd847\" (UID: \"b8ac1cca-4ebd-474c-b32f-a3595a4cd847\") "
Oct 30 21:02:48.533398 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.533027    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc0f5352-5c44-4c35-adf5-3fffab06b644-home\" (UniqueName: \"kubernetes.io/empty-dir/bc0f5352-5c44-4c35-adf5-3fffab06b644-home\") pod \"bc0f5352-5c44-4c35-adf5-3fffab06b644\" (UID: \"bc0f5352-5c44-4c35-adf5-3fffab06b644\") "
Oct 30 21:02:48.553569 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.551870    4521 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00b6adb9-3733-4ea4-9a26-f3b29991001a-istio-envoy\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:48.553569 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.551905    4521 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1fa6900c-eee2-4f00-b0ec-248def48dfc4-dshm\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:48.553569 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.551920    4521 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xljl7\" (UniqueName: \"kubernetes.io/projected/1fa6900c-eee2-4f00-b0ec-248def48dfc4-kube-api-access-xljl7\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:48.553569 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.551935    4521 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tshfg\" (UniqueName: \"kubernetes.io/projected/bc0f5352-5c44-4c35-adf5-3fffab06b644-kube-api-access-tshfg\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:48.553569 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.551951    4521 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8ac1cca-4ebd-474c-b32f-a3595a4cd847-tmp-dir\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:48.553569 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.551964    4521 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc0f5352-5c44-4c35-adf5-3fffab06b644-tmp-dir\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:48.553569 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.551976    4521 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00b6adb9-3733-4ea4-9a26-f3b29991001a-istio-token\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:48.553569 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.551989    4521 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8ac1cca-4ebd-474c-b32f-a3595a4cd847-model-cache\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:48.553569 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.552004    4521 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmrvp\" (UniqueName: \"kubernetes.io/projected/00b6adb9-3733-4ea4-9a26-f3b29991001a-kube-api-access-xmrvp\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:48.553569 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.552018    4521 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00b6adb9-3733-4ea4-9a26-f3b29991001a-istio-podinfo\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:48.553569 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.552033    4521 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1fa6900c-eee2-4f00-b0ec-248def48dfc4-tls-certs\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:48.578274 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.559639    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc0f5352-5c44-4c35-adf5-3fffab06b644-home" pod "bc0f5352-5c44-4c35-adf5-3fffab06b644" (UID: "bc0f5352-5c44-4c35-adf5-3fffab06b644"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:02:48.578274 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.559870    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6751cf5-7df8-4ffa-b6c8-8b0c4f16ba2a-model-cache" pod "d6751cf5-7df8-4ffa-b6c8-8b0c4f16ba2a" (UID: "d6751cf5-7df8-4ffa-b6c8-8b0c4f16ba2a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:02:48.578274 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.560084    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8ac1cca-4ebd-474c-b32f-a3595a4cd847-home" pod "b8ac1cca-4ebd-474c-b32f-a3595a4cd847" (UID: "b8ac1cca-4ebd-474c-b32f-a3595a4cd847"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:02:48.579212 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.578901    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6751cf5-7df8-4ffa-b6c8-8b0c4f16ba2a-home" pod "d6751cf5-7df8-4ffa-b6c8-8b0c4f16ba2a" (UID: "d6751cf5-7df8-4ffa-b6c8-8b0c4f16ba2a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:02:48.579212 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.579172    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc0f5352-5c44-4c35-adf5-3fffab06b644-model-cache" pod "bc0f5352-5c44-4c35-adf5-3fffab06b644" (UID: "bc0f5352-5c44-4c35-adf5-3fffab06b644"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:02:48.579528 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.579491    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6751cf5-7df8-4ffa-b6c8-8b0c4f16ba2a-tmp-dir" pod "d6751cf5-7df8-4ffa-b6c8-8b0c4f16ba2a" (UID: "d6751cf5-7df8-4ffa-b6c8-8b0c4f16ba2a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:02:48.580918 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.580779    4521 scope.go:122] "RemoveContainer" containerID="792caf04aadf9b69e35ec7d46269a59947e1107749f2465ea6a82633fae13abd"
Oct 30 21:02:48.603585 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.603522    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8ac1cca-4ebd-474c-b32f-a3595a4cd847-kube-api-access-bstjz" pod "b8ac1cca-4ebd-474c-b32f-a3595a4cd847" (UID: "b8ac1cca-4ebd-474c-b32f-a3595a4cd847"). InnerVolumeSpecName "kube-api-access-bstjz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:02:48.613952 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.613868    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8ac1cca-4ebd-474c-b32f-a3595a4cd847-tls-certs" pod "b8ac1cca-4ebd-474c-b32f-a3595a4cd847" (UID: "b8ac1cca-4ebd-474c-b32f-a3595a4cd847"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:02:48.648267 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.647744    4521 patch_prober.go:29] interesting pod/e2e-worker-w5-65968b-openshift-default-68896fcddc-9zp68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:02:48.648267 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.647808    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-9zp68" podUID="65e4dd7b-506f-4350-a82c-e35e61706595" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:02:48.690368 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.689189    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d6751cf5-7df8-4ffa-b6c8-8b0c4f16ba2a-tls-certs\" (UniqueName: \"kubernetes.io/secret/d6751cf5-7df8-4ffa-b6c8-8b0c4f16ba2a-tls-certs\") pod \"d6751cf5-7df8-4ffa-b6c8-8b0c4f16ba2a\" (UID: \"d6751cf5-7df8-4ffa-b6c8-8b0c4f16ba2a\") "
Oct 30 21:02:48.690368 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.689284    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6751cf5-7df8-4ffa-b6c8-8b0c4f16ba2a-kube-api-access-7vjbg\" (UniqueName: \"kubernetes.io/projected/d6751cf5-7df8-4ffa-b6c8-8b0c4f16ba2a-kube-api-access-7vjbg\") pod \"d6751cf5-7df8-4ffa-b6c8-8b0c4f16ba2a\" (UID: \"d6751cf5-7df8-4ffa-b6c8-8b0c4f16ba2a\") "
Oct 30 21:02:48.690368 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.689381    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6751cf5-7df8-4ffa-b6c8-8b0c4f16ba2a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6751cf5-7df8-4ffa-b6c8-8b0c4f16ba2a-dshm\") pod \"d6751cf5-7df8-4ffa-b6c8-8b0c4f16ba2a\" (UID: \"d6751cf5-7df8-4ffa-b6c8-8b0c4f16ba2a\") "
Oct 30 21:02:48.699975 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.698777    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a8f94/distinct-2-w3-5a8f94-kserve-ffcc6449b-hv48k" podUID="bc0f5352-5c44-4c35-adf5-3fffab06b644" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:02:48.864676 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.779820    4521 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6751cf5-7df8-4ffa-b6c8-8b0c4f16ba2a-home\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:48.864676 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.781001    4521 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc0f5352-5c44-4c35-adf5-3fffab06b644-model-cache\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:48.864676 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.781203    4521 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8ac1cca-4ebd-474c-b32f-a3595a4cd847-tls-certs\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:48.864676 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.781226    4521 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6751cf5-7df8-4ffa-b6c8-8b0c4f16ba2a-model-cache\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:48.864676 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.781245    4521 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8ac1cca-4ebd-474c-b32f-a3595a4cd847-home\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:48.864676 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.781268    4521 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc0f5352-5c44-4c35-adf5-3fffab06b644-home\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:48.864676 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.781284    4521 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bstjz\" (UniqueName: \"kubernetes.io/projected/b8ac1cca-4ebd-474c-b32f-a3595a4cd847-kube-api-access-bstjz\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:48.864676 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.781301    4521 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6751cf5-7df8-4ffa-b6c8-8b0c4f16ba2a-tmp-dir\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:48.864676 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.795815    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc0f5352-5c44-4c35-adf5-3fffab06b644-dshm" pod "bc0f5352-5c44-4c35-adf5-3fffab06b644" (UID: "bc0f5352-5c44-4c35-adf5-3fffab06b644"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:02:48.864676 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.795823    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc0f5352-5c44-4c35-adf5-3fffab06b644-tls-certs" pod "bc0f5352-5c44-4c35-adf5-3fffab06b644" (UID: "bc0f5352-5c44-4c35-adf5-3fffab06b644"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:02:48.864676 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.803104    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8ac1cca-4ebd-474c-b32f-a3595a4cd847-dshm" pod "b8ac1cca-4ebd-474c-b32f-a3595a4cd847" (UID: "b8ac1cca-4ebd-474c-b32f-a3595a4cd847"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:02:48.864676 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.805395    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6751cf5-7df8-4ffa-b6c8-8b0c4f16ba2a-kube-api-access-7vjbg" pod "d6751cf5-7df8-4ffa-b6c8-8b0c4f16ba2a" (UID: "d6751cf5-7df8-4ffa-b6c8-8b0c4f16ba2a"). InnerVolumeSpecName "kube-api-access-7vjbg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:02:48.864676 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.805666    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6751cf5-7df8-4ffa-b6c8-8b0c4f16ba2a-tls-certs" pod "d6751cf5-7df8-4ffa-b6c8-8b0c4f16ba2a" (UID: "d6751cf5-7df8-4ffa-b6c8-8b0c4f16ba2a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:02:48.864676 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.809091    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6751cf5-7df8-4ffa-b6c8-8b0c4f16ba2a-dshm" pod "d6751cf5-7df8-4ffa-b6c8-8b0c4f16ba2a" (UID: "d6751cf5-7df8-4ffa-b6c8-8b0c4f16ba2a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:02:48.904557 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.898069    4521 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8ac1cca-4ebd-474c-b32f-a3595a4cd847-dshm\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:48.904557 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.898101    4521 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6751cf5-7df8-4ffa-b6c8-8b0c4f16ba2a-tls-certs\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:48.904557 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.898117    4521 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc0f5352-5c44-4c35-adf5-3fffab06b644-dshm\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:48.904557 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.898133    4521 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vjbg\" (UniqueName: \"kubernetes.io/projected/d6751cf5-7df8-4ffa-b6c8-8b0c4f16ba2a-kube-api-access-7vjbg\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:48.904557 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.898154    4521 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6751cf5-7df8-4ffa-b6c8-8b0c4f16ba2a-dshm\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:48.904557 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.898170    4521 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc0f5352-5c44-4c35-adf5-3fffab06b644-tls-certs\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:48.948551 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.948473    4521 patch_prober.go:29] interesting pod/e2e-worker-w5-65968b-openshift-default-68896fcddc-sbpfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:02:48.948698 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.948554    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-sbpfk" podUID="7a5f91bb-7229-454f-a006-0d768f8879f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:02:48.981553 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.981519    4521 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24ccc37d-027a-4097-8da5-04df417aef94" path="/var/lib/kubelet/pods/24ccc37d-027a-4097-8da5-04df417aef94/volumes"
Oct 30 21:02:48.985731 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:48.985705    4521 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-65968b/premium-w5-65968b-kserve-67977485f7-c9jpv"]
Oct 30 21:02:49.614437 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:49.614404    4521 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-65968b/premium-w5-65968b-kserve-67977485f7-c9jpv"]
Oct 30 21:02:49.614948 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:49.614930    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-617df4d2-t4ck6"]
Oct 30 21:02:49.615079 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:49.615067    4521 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-c2d1eb/premium-w4-c2d1eb-kserve-8696fbd9b9-mxqhm"]
Oct 30 21:02:49.615165 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:49.311569    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-hmwtj"
Oct 30 21:02:49.630350 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:49.120933    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-65968b/distinct-2-w5-65968b-kserve-6d65c85f6c-dz5r7" podUID="d6751cf5-7df8-4ffa-b6c8-8b0c4f16ba2a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:02:49.630926 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:49.306445    4521 patch_prober.go:29] interesting pod/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-m9wn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:02:49.631108 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:49.631083    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-m9wn4" podUID="9dc1356e-e12d-49f9-822e-f1ccf4fdaa3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:02:49.631517 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:49.532882    4521 patch_prober.go:29] interesting pod/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-nvwt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:02:49.631654 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:49.631633    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-nvwt8" podUID="6cff2b61-dcbd-4450-b776-43eff57a4fd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:02:49.709004 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:49.708968    4521 patch_prober.go:29] interesting pod/e2e-worker-w5-65968b-openshift-default-68896fcddc-9zp68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:02:49.709293 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:49.709250    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-9zp68" podUID="65e4dd7b-506f-4350-a82c-e35e61706595" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:02:49.795635 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:49.795598    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f6f47707-ed2c-4860-a13a-b1a5d9d1bbe3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6f47707-ed2c-4860-a13a-b1a5d9d1bbe3-istio-podinfo\") pod \"f6f47707-ed2c-4860-a13a-b1a5d9d1bbe3\" (UID: \"f6f47707-ed2c-4860-a13a-b1a5d9d1bbe3\") "
Oct 30 21:02:49.796793 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:49.795947    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6f47707-ed2c-4860-a13a-b1a5d9d1bbe3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6f47707-ed2c-4860-a13a-b1a5d9d1bbe3-workload-socket\") pod \"f6f47707-ed2c-4860-a13a-b1a5d9d1bbe3\" (UID: \"f6f47707-ed2c-4860-a13a-b1a5d9d1bbe3\") "
Oct 30 21:02:49.797007 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:49.796981    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6f47707-ed2c-4860-a13a-b1a5d9d1bbe3-workload-socket" pod "f6f47707-ed2c-4860-a13a-b1a5d9d1bbe3" (UID: "f6f47707-ed2c-4860-a13a-b1a5d9d1bbe3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:02:49.798052 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:49.797985    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72ee85f3-2hq26" event={"ID":"1e857e76-3d48-4e5b-a8b5-830cfac13f90","Type":"ContainerStarted","Data":"36e13db3282053822160ff0d396939c7cf6fe65571880e8d18973180a449615e"}
Oct 30 21:02:49.800190 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:49.800164    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-617df4d2-t4ck6" event={"ID":"75d8f639-8cee-4be3-af28-eb7886627983","Type":"ContainerStarted","Data":"e86a3f1c53d6bb55b7d96466837556c6fb37e0f04a6b340098c84190f1d51014"}
Oct 30 21:02:49.810941 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:49.810905    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-hmwtj" event={"ID":"f6f47707-ed2c-4860-a13a-b1a5d9d1bbe3","Type":"ContainerDied","Data":"b31b2b9d5bb2a2d3056bf95aafc88f4cbd316eb4b9143c6faf29d592f54df223"}
Oct 30 21:02:49.811143 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:49.811126    4521 scope.go:122] "RemoveContainer" containerID="8b8bd567c67d21ef122a8fd6141d1c042d7bfca248ef665aa84d0e993110b661"
Oct 30 21:02:49.815965 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:49.815945    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-hmwtj"
Oct 30 21:02:49.817769 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:49.817752    4521 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-c2d1eb/premium-w4-c2d1eb-kserve-8696fbd9b9-mxqhm"]
Oct 30 21:02:49.839712 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:49.839679    4521 generic.go:356] "Generic (PLEG): container finished" podID="7d10d07e-a60e-4c6a-b067-66234939fa57" containerID="5cf7f7a8b1f335911c58c99ac404c84165377fe9f2d6e87979e4ec7dc0d1b324" exitCode=0
Oct 30 21:02:49.839969 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:49.839949    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-408918eb-f26js" event={"ID":"7d10d07e-a60e-4c6a-b067-66234939fa57","Type":"ContainerDied","Data":"5cf7f7a8b1f335911c58c99ac404c84165377fe9f2d6e87979e4ec7dc0d1b324"}
Oct 30 21:02:49.855711 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:49.855664    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-65968b/distinct-2-w5-65968b-kserve-6d65c85f6c-dz5r7"
Oct 30 21:02:49.882396 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:49.865523    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-5a8f94/distinct-2-w3-5a8f94-kserve-ffcc6449b-hv48k"
Oct 30 21:02:49.902402 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:49.865570    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-5a8f94/distinct-w3-5a8f94-kserve-5f8b544d59-mf95j"
Oct 30 21:02:49.904701 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:49.904671    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6f47707-ed2c-4860-a13a-b1a5d9d1bbe3-istio-token\" (UniqueName: \"kubernetes.io/projected/f6f47707-ed2c-4860-a13a-b1a5d9d1bbe3-istio-token\") pod \"f6f47707-ed2c-4860-a13a-b1a5d9d1bbe3\" (UID: \"f6f47707-ed2c-4860-a13a-b1a5d9d1bbe3\") "
Oct 30 21:02:49.904981 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:49.904961    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6f47707-ed2c-4860-a13a-b1a5d9d1bbe3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6f47707-ed2c-4860-a13a-b1a5d9d1bbe3-workload-certs\") pod \"f6f47707-ed2c-4860-a13a-b1a5d9d1bbe3\" (UID: \"f6f47707-ed2c-4860-a13a-b1a5d9d1bbe3\") "
Oct 30 21:02:49.905168 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:49.905154    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6f47707-ed2c-4860-a13a-b1a5d9d1bbe3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6f47707-ed2c-4860-a13a-b1a5d9d1bbe3-istiod-ca-cert\") pod \"f6f47707-ed2c-4860-a13a-b1a5d9d1bbe3\" (UID: \"f6f47707-ed2c-4860-a13a-b1a5d9d1bbe3\") "
Oct 30 21:02:49.905295 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:49.905280    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6f47707-ed2c-4860-a13a-b1a5d9d1bbe3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6f47707-ed2c-4860-a13a-b1a5d9d1bbe3-istio-data\") pod \"f6f47707-ed2c-4860-a13a-b1a5d9d1bbe3\" (UID: \"f6f47707-ed2c-4860-a13a-b1a5d9d1bbe3\") "
Oct 30 21:02:49.921346 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:49.905477    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6f47707-ed2c-4860-a13a-b1a5d9d1bbe3-kube-api-access-288zg\" (UniqueName: \"kubernetes.io/projected/f6f47707-ed2c-4860-a13a-b1a5d9d1bbe3-kube-api-access-288zg\") pod \"f6f47707-ed2c-4860-a13a-b1a5d9d1bbe3\" (UID: \"f6f47707-ed2c-4860-a13a-b1a5d9d1bbe3\") "
Oct 30 21:02:49.923926 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:49.923888    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6f47707-ed2c-4860-a13a-b1a5d9d1bbe3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6f47707-ed2c-4860-a13a-b1a5d9d1bbe3-credential-socket\") pod \"f6f47707-ed2c-4860-a13a-b1a5d9d1bbe3\" (UID: \"f6f47707-ed2c-4860-a13a-b1a5d9d1bbe3\") "
Oct 30 21:02:49.924179 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:49.924149    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6f47707-ed2c-4860-a13a-b1a5d9d1bbe3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6f47707-ed2c-4860-a13a-b1a5d9d1bbe3-istio-envoy\") pod \"f6f47707-ed2c-4860-a13a-b1a5d9d1bbe3\" (UID: \"f6f47707-ed2c-4860-a13a-b1a5d9d1bbe3\") "
Oct 30 21:02:49.948562 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:49.943136    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6f47707-ed2c-4860-a13a-b1a5d9d1bbe3-credential-socket" pod "f6f47707-ed2c-4860-a13a-b1a5d9d1bbe3" (UID: "f6f47707-ed2c-4860-a13a-b1a5d9d1bbe3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:02:49.979754 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:49.970610    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6f47707-ed2c-4860-a13a-b1a5d9d1bbe3-workload-certs" pod "f6f47707-ed2c-4860-a13a-b1a5d9d1bbe3" (UID: "f6f47707-ed2c-4860-a13a-b1a5d9d1bbe3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:02:49.986525 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:49.986492    4521 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6f47707-ed2c-4860-a13a-b1a5d9d1bbe3-workload-socket\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:49.998557 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:49.998523    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6f47707-ed2c-4860-a13a-b1a5d9d1bbe3-istiod-ca-cert" pod "f6f47707-ed2c-4860-a13a-b1a5d9d1bbe3" (UID: "f6f47707-ed2c-4860-a13a-b1a5d9d1bbe3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:02:50.006599 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:50.006478    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6f47707-ed2c-4860-a13a-b1a5d9d1bbe3-istio-data" pod "f6f47707-ed2c-4860-a13a-b1a5d9d1bbe3" (UID: "f6f47707-ed2c-4860-a13a-b1a5d9d1bbe3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:02:50.012578 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:50.007047    4521 patch_prober.go:29] interesting pod/e2e-worker-w5-65968b-openshift-default-68896fcddc-sbpfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:02:50.012578 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:50.007109    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-sbpfk" podUID="7a5f91bb-7229-454f-a006-0d768f8879f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:02:50.012578 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:50.007678    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6f47707-ed2c-4860-a13a-b1a5d9d1bbe3-kube-api-access-288zg" pod "f6f47707-ed2c-4860-a13a-b1a5d9d1bbe3" (UID: "f6f47707-ed2c-4860-a13a-b1a5d9d1bbe3"). InnerVolumeSpecName "kube-api-access-288zg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:02:50.033478 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:50.033406    4521 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-p55rm"]
Oct 30 21:02:50.047553 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:50.047498    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6f47707-ed2c-4860-a13a-b1a5d9d1bbe3-istio-envoy" pod "f6f47707-ed2c-4860-a13a-b1a5d9d1bbe3" (UID: "f6f47707-ed2c-4860-a13a-b1a5d9d1bbe3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:02:50.060446 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:50.060209    4521 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-p55rm"]
Oct 30 21:02:50.060905 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:50.060859    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f6f47707-ed2c-4860-a13a-b1a5d9d1bbe3-istio-podinfo" pod "f6f47707-ed2c-4860-a13a-b1a5d9d1bbe3" (UID: "f6f47707-ed2c-4860-a13a-b1a5d9d1bbe3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:02:50.069695 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:50.069639    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6f47707-ed2c-4860-a13a-b1a5d9d1bbe3-istio-token" pod "f6f47707-ed2c-4860-a13a-b1a5d9d1bbe3" (UID: "f6f47707-ed2c-4860-a13a-b1a5d9d1bbe3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:02:50.073211 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:50.073139    4521 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-5a8f94/simulator-w3-5a8f94-kserve-77cc88cc89-n4sl6"]
Oct 30 21:02:50.088658 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:50.088476    4521 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6f47707-ed2c-4860-a13a-b1a5d9d1bbe3-istiod-ca-cert\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:50.088658 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:50.088512    4521 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6f47707-ed2c-4860-a13a-b1a5d9d1bbe3-istio-data\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:50.088658 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:50.088527    4521 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-288zg\" (UniqueName: \"kubernetes.io/projected/f6f47707-ed2c-4860-a13a-b1a5d9d1bbe3-kube-api-access-288zg\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:50.088658 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:50.088544    4521 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6f47707-ed2c-4860-a13a-b1a5d9d1bbe3-credential-socket\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:50.088658 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:50.088559    4521 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6f47707-ed2c-4860-a13a-b1a5d9d1bbe3-istio-envoy\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:50.088658 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:50.088573    4521 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6f47707-ed2c-4860-a13a-b1a5d9d1bbe3-istio-podinfo\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:50.088658 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:50.088586    4521 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6f47707-ed2c-4860-a13a-b1a5d9d1bbe3-istio-token\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:50.088658 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:50.088600    4521 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6f47707-ed2c-4860-a13a-b1a5d9d1bbe3-workload-certs\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:50.089940 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:50.089870    4521 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-5a8f94/simulator-w3-5a8f94-kserve-77cc88cc89-n4sl6"]
Oct 30 21:02:50.099831 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:50.099735    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-72ee85f3-2hq26"]
Oct 30 21:02:50.181643 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:50.162181    4521 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-tqd8s"]
Oct 30 21:02:50.181643 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:50.172379    4521 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-tqd8s"]
Oct 30 21:02:50.233349 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:50.226618    4521 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-65968b/distinct-2-w5-65968b-kserve-6d65c85f6c-dz5r7"]
Oct 30 21:02:50.237764 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:50.237736    4521 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-65968b/distinct-2-w5-65968b-kserve-6d65c85f6c-dz5r7"]
Oct 30 21:02:50.238009 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:50.237992    4521 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-5a8f94/distinct-w3-5a8f94-kserve-5f8b544d59-mf95j"]
Oct 30 21:02:50.246791 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:50.246718    4521 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-5a8f94/distinct-w3-5a8f94-kserve-5f8b544d59-mf95j"]
Oct 30 21:02:50.258196 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:50.257399    4521 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-5a8f94/distinct-2-w3-5a8f94-kserve-ffcc6449b-hv48k"]
Oct 30 21:02:50.265026 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:50.264958    4521 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-5a8f94/distinct-2-w3-5a8f94-kserve-ffcc6449b-hv48k"]
Oct 30 21:02:50.295573 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:50.294002    4521 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-nvwt8"
Oct 30 21:02:50.301381 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:50.300775    4521 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nsppg"]
Oct 30 21:02:50.303648 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:50.302143    4521 cpu_manager.go:414] "containerMap: removing container" podUID="4bb0f898-b77f-419d-bff4-6413e82ef51d" containerName="main"
Oct 30 21:02:50.303648 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:50.302170    4521 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bb0f898-b77f-419d-bff4-6413e82ef51d" containerName="main"
Oct 30 21:02:50.303648 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:50.302188    4521 cpu_manager.go:414] "containerMap: removing container" podUID="f6f47707-ed2c-4860-a13a-b1a5d9d1bbe3" containerName="istio-proxy"
Oct 30 21:02:50.303648 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:50.302196    4521 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6f47707-ed2c-4860-a13a-b1a5d9d1bbe3" containerName="istio-proxy"
Oct 30 21:02:50.303648 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:50.302205    4521 cpu_manager.go:414] "containerMap: removing container" podUID="d6751cf5-7df8-4ffa-b6c8-8b0c4f16ba2a" containerName="main"
Oct 30 21:02:50.303648 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:50.302212    4521 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6751cf5-7df8-4ffa-b6c8-8b0c4f16ba2a" containerName="main"
Oct 30 21:02:50.303648 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:50.302229    4521 cpu_manager.go:414] "containerMap: removing container" podUID="1fa6900c-eee2-4f00-b0ec-248def48dfc4" containerName="main"
Oct 30 21:02:50.303648 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:50.302236    4521 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fa6900c-eee2-4f00-b0ec-248def48dfc4" containerName="main"
Oct 30 21:02:50.303648 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:50.302246    4521 cpu_manager.go:414] "containerMap: removing container" podUID="b8ac1cca-4ebd-474c-b32f-a3595a4cd847" containerName="main"
Oct 30 21:02:50.303648 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:50.302253    4521 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8ac1cca-4ebd-474c-b32f-a3595a4cd847" containerName="main"
Oct 30 21:02:50.303648 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:50.302275    4521 cpu_manager.go:414] "containerMap: removing container" podUID="92b2dd65-8a83-436d-afa8-50e5f0d11fa0" containerName="main"
Oct 30 21:02:50.303648 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:50.302282    4521 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92b2dd65-8a83-436d-afa8-50e5f0d11fa0" containerName="main"
Oct 30 21:02:50.303648 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:50.302300    4521 cpu_manager.go:414] "containerMap: removing container" podUID="baad1994-acb5-4aae-bb65-a5e8a5b8afdc" containerName="istio-proxy"
Oct 30 21:02:50.303648 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:50.302308    4521 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="baad1994-acb5-4aae-bb65-a5e8a5b8afdc" containerName="istio-proxy"
Oct 30 21:02:50.303648 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:50.302337    4521 cpu_manager.go:414] "containerMap: removing container" podUID="bc0f5352-5c44-4c35-adf5-3fffab06b644" containerName="main"
Oct 30 21:02:50.303648 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:50.302345    4521 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc0f5352-5c44-4c35-adf5-3fffab06b644" containerName="main"
Oct 30 21:02:50.303648 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:50.302366    4521 cpu_manager.go:414] "containerMap: removing container" podUID="24ccc37d-027a-4097-8da5-04df417aef94" containerName="main"
Oct 30 21:02:50.303648 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:50.302373    4521 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24ccc37d-027a-4097-8da5-04df417aef94" containerName="main"
Oct 30 21:02:50.303648 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:50.302385    4521 cpu_manager.go:414] "containerMap: removing container" podUID="00b6adb9-3733-4ea4-9a26-f3b29991001a" containerName="istio-proxy"
Oct 30 21:02:50.303648 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:50.302393    4521 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00b6adb9-3733-4ea4-9a26-f3b29991001a" containerName="istio-proxy"
Oct 30 21:02:50.303648 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:50.302568    4521 memory_manager.go:365] "RemoveStaleState removing state" podUID="24ccc37d-027a-4097-8da5-04df417aef94" containerName="main"
Oct 30 21:02:50.303648 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:50.302584    4521 memory_manager.go:365] "RemoveStaleState removing state" podUID="00b6adb9-3733-4ea4-9a26-f3b29991001a" containerName="istio-proxy"
Oct 30 21:02:50.303648 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:50.302612    4521 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bb0f898-b77f-419d-bff4-6413e82ef51d" containerName="main"
Oct 30 21:02:50.303648 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:50.302621    4521 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6f47707-ed2c-4860-a13a-b1a5d9d1bbe3" containerName="istio-proxy"
Oct 30 21:02:50.303648 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:50.302629    4521 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6751cf5-7df8-4ffa-b6c8-8b0c4f16ba2a" containerName="main"
Oct 30 21:02:50.303648 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:50.302640    4521 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fa6900c-eee2-4f00-b0ec-248def48dfc4" containerName="main"
Oct 30 21:02:50.303648 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:50.302650    4521 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8ac1cca-4ebd-474c-b32f-a3595a4cd847" containerName="main"
Oct 30 21:02:50.303648 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:50.302660    4521 memory_manager.go:365] "RemoveStaleState removing state" podUID="92b2dd65-8a83-436d-afa8-50e5f0d11fa0" containerName="main"
Oct 30 21:02:50.303648 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:50.302671    4521 memory_manager.go:365] "RemoveStaleState removing state" podUID="baad1994-acb5-4aae-bb65-a5e8a5b8afdc" containerName="istio-proxy"
Oct 30 21:02:50.303648 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:50.302681    4521 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc0f5352-5c44-4c35-adf5-3fffab06b644" containerName="main"
Oct 30 21:02:50.317386 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:50.309182    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nsppg"
Oct 30 21:02:50.367172 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:50.367089    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nsppg"]
Oct 30 21:02:50.375833 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:50.373892    4521 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Oct 30 21:02:50.375833 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:50.373972    4521 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-fnh57\""
Oct 30 21:02:50.375833 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:50.374031    4521 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Oct 30 21:02:50.375833 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:50.374073    4521 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Oct 30 21:02:50.393256 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:50.393220    4521 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-m9wn4"
Oct 30 21:02:50.421403 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:50.421363    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47a667bb-4dac-484a-9f1b-01ef17c79710-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nsppg\" (UID: \"47a667bb-4dac-484a-9f1b-01ef17c79710\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nsppg"
Oct 30 21:02:50.421773 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:50.421748    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/47a667bb-4dac-484a-9f1b-01ef17c79710-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nsppg\" (UID: \"47a667bb-4dac-484a-9f1b-01ef17c79710\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nsppg"
Oct 30 21:02:50.422027 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:50.422004    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/47a667bb-4dac-484a-9f1b-01ef17c79710-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nsppg\" (UID: \"47a667bb-4dac-484a-9f1b-01ef17c79710\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nsppg"
Oct 30 21:02:50.422249 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:50.422229    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/47a667bb-4dac-484a-9f1b-01ef17c79710-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nsppg\" (UID: \"47a667bb-4dac-484a-9f1b-01ef17c79710\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nsppg"
Oct 30 21:02:50.422464 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:50.422443    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/47a667bb-4dac-484a-9f1b-01ef17c79710-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nsppg\" (UID: \"47a667bb-4dac-484a-9f1b-01ef17c79710\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nsppg"
Oct 30 21:02:50.422676 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:50.422656    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/47a667bb-4dac-484a-9f1b-01ef17c79710-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nsppg\" (UID: \"47a667bb-4dac-484a-9f1b-01ef17c79710\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nsppg"
Oct 30 21:02:50.423152 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:50.422910    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnvs8\" (UniqueName: \"kubernetes.io/projected/47a667bb-4dac-484a-9f1b-01ef17c79710-kube-api-access-xnvs8\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nsppg\" (UID: \"47a667bb-4dac-484a-9f1b-01ef17c79710\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nsppg"
Oct 30 21:02:50.441434 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:50.438404    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-m9wn4"
Oct 30 21:02:50.444868 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:50.444840    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-nvwt8"
Oct 30 21:02:50.512349 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:50.508378    4521 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-hmwtj"]
Oct 30 21:02:50.549404 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:50.531209    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/47a667bb-4dac-484a-9f1b-01ef17c79710-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nsppg\" (UID: \"47a667bb-4dac-484a-9f1b-01ef17c79710\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nsppg"
Oct 30 21:02:50.549404 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:50.546851    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/47a667bb-4dac-484a-9f1b-01ef17c79710-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nsppg\" (UID: \"47a667bb-4dac-484a-9f1b-01ef17c79710\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nsppg"
Oct 30 21:02:50.549404 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:50.546927    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnvs8\" (UniqueName: \"kubernetes.io/projected/47a667bb-4dac-484a-9f1b-01ef17c79710-kube-api-access-xnvs8\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nsppg\" (UID: \"47a667bb-4dac-484a-9f1b-01ef17c79710\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nsppg"
Oct 30 21:02:50.549404 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:50.547032    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47a667bb-4dac-484a-9f1b-01ef17c79710-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nsppg\" (UID: \"47a667bb-4dac-484a-9f1b-01ef17c79710\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nsppg"
Oct 30 21:02:50.549404 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:50.547229    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/47a667bb-4dac-484a-9f1b-01ef17c79710-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nsppg\" (UID: \"47a667bb-4dac-484a-9f1b-01ef17c79710\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nsppg"
Oct 30 21:02:50.549404 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:50.547407    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/47a667bb-4dac-484a-9f1b-01ef17c79710-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nsppg\" (UID: \"47a667bb-4dac-484a-9f1b-01ef17c79710\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nsppg"
Oct 30 21:02:50.549404 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:50.547519    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/47a667bb-4dac-484a-9f1b-01ef17c79710-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nsppg\" (UID: \"47a667bb-4dac-484a-9f1b-01ef17c79710\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nsppg"
Oct 30 21:02:50.549404 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:50.541716    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/47a667bb-4dac-484a-9f1b-01ef17c79710-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nsppg\" (UID: \"47a667bb-4dac-484a-9f1b-01ef17c79710\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nsppg"
Oct 30 21:02:50.549404 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:50.549141    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/47a667bb-4dac-484a-9f1b-01ef17c79710-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nsppg\" (UID: \"47a667bb-4dac-484a-9f1b-01ef17c79710\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nsppg"
Oct 30 21:02:50.554694 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:50.554624    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/47a667bb-4dac-484a-9f1b-01ef17c79710-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nsppg\" (UID: \"47a667bb-4dac-484a-9f1b-01ef17c79710\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nsppg"
Oct 30 21:02:50.554921 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:50.554901    4521 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-32a969-openshift-default-8945d87fc-hmwtj"]
Oct 30 21:02:50.555058 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:50.555034    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/47a667bb-4dac-484a-9f1b-01ef17c79710-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nsppg\" (UID: \"47a667bb-4dac-484a-9f1b-01ef17c79710\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nsppg"
Oct 30 21:02:50.555985 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:50.555937    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/47a667bb-4dac-484a-9f1b-01ef17c79710-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nsppg\" (UID: \"47a667bb-4dac-484a-9f1b-01ef17c79710\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nsppg"
Oct 30 21:02:50.578319 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:50.578287    4521 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-sbpfk"
Oct 30 21:02:50.587625 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:50.585930    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47a667bb-4dac-484a-9f1b-01ef17c79710-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nsppg\" (UID: \"47a667bb-4dac-484a-9f1b-01ef17c79710\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nsppg"
Oct 30 21:02:50.588557 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:50.588480    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnvs8\" (UniqueName: \"kubernetes.io/projected/47a667bb-4dac-484a-9f1b-01ef17c79710-kube-api-access-xnvs8\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nsppg\" (UID: \"47a667bb-4dac-484a-9f1b-01ef17c79710\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nsppg"
Oct 30 21:02:50.590070 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:50.590006    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-sbpfk"
Oct 30 21:02:50.656221 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:50.656181    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nsppg"
Oct 30 21:02:50.666819 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:50.666792    4521 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-9zp68"
Oct 30 21:02:50.671794 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:50.671720    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-9zp68"
Oct 30 21:02:50.793427 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:50.793086    4521 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00b6adb9-3733-4ea4-9a26-f3b29991001a" path="/var/lib/kubelet/pods/00b6adb9-3733-4ea4-9a26-f3b29991001a/volumes"
Oct 30 21:02:50.793728 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:50.793706    4521 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1fa6900c-eee2-4f00-b0ec-248def48dfc4" path="/var/lib/kubelet/pods/1fa6900c-eee2-4f00-b0ec-248def48dfc4/volumes"
Oct 30 21:02:50.794190 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:50.794171    4521 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4bb0f898-b77f-419d-bff4-6413e82ef51d" path="/var/lib/kubelet/pods/4bb0f898-b77f-419d-bff4-6413e82ef51d/volumes"
Oct 30 21:02:50.794962 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:50.794934    4521 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92b2dd65-8a83-436d-afa8-50e5f0d11fa0" path="/var/lib/kubelet/pods/92b2dd65-8a83-436d-afa8-50e5f0d11fa0/volumes"
Oct 30 21:02:50.795390 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:50.795365    4521 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8ac1cca-4ebd-474c-b32f-a3595a4cd847" path="/var/lib/kubelet/pods/b8ac1cca-4ebd-474c-b32f-a3595a4cd847/volumes"
Oct 30 21:02:50.795788 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:50.795775    4521 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="baad1994-acb5-4aae-bb65-a5e8a5b8afdc" path="/var/lib/kubelet/pods/baad1994-acb5-4aae-bb65-a5e8a5b8afdc/volumes"
Oct 30 21:02:50.796507 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:50.796491    4521 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc0f5352-5c44-4c35-adf5-3fffab06b644" path="/var/lib/kubelet/pods/bc0f5352-5c44-4c35-adf5-3fffab06b644/volumes"
Oct 30 21:02:50.796976 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:50.796963    4521 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6751cf5-7df8-4ffa-b6c8-8b0c4f16ba2a" path="/var/lib/kubelet/pods/d6751cf5-7df8-4ffa-b6c8-8b0c4f16ba2a/volumes"
Oct 30 21:02:50.797462 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:50.797450    4521 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6f47707-ed2c-4860-a13a-b1a5d9d1bbe3" path="/var/lib/kubelet/pods/f6f47707-ed2c-4860-a13a-b1a5d9d1bbe3/volumes"
Oct 30 21:02:50.880792 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:50.880699    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72ee85f3-2hq26" event={"ID":"1e857e76-3d48-4e5b-a8b5-830cfac13f90","Type":"ContainerStarted","Data":"9e011a83e7dad15b724390446e366ea7ab9f3a5ba297591c56164b3b6ad78ab2"}
Oct 30 21:02:50.885640 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:50.884771    4521 generic.go:356] "Generic (PLEG): container finished" podID="75d8f639-8cee-4be3-af28-eb7886627983" containerID="a8372a359a760d7091e41da2dfbfdc93b44450e5db6246bbcba52fd64c2473b0" exitCode=0
Oct 30 21:02:50.885906 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:50.885851    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-617df4d2-t4ck6" event={"ID":"75d8f639-8cee-4be3-af28-eb7886627983","Type":"ContainerDied","Data":"a8372a359a760d7091e41da2dfbfdc93b44450e5db6246bbcba52fd64c2473b0"}
Oct 30 21:02:50.902892 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:50.902673    4521 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72ee85f3-2hq26" podStartSLOduration=7.902658574 podStartE2EDuration="7.902658574s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:02:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:02:50.902641507 +0000 UTC m=+1438.248262746" watchObservedRunningTime="2026-10-30 21:02:50.902658574 +0000 UTC m=+1438.248279822"
Oct 30 21:02:51.005829 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:51.005781    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nsppg"]
Oct 30 21:02:51.037352 ip-10-0-131-125 kubenswrapper[4521]: W1030 21:02:51.030375    4521 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47a667bb_4dac_484a_9f1b_01ef17c79710.slice/crio-43efee53b6c8cc9e2eedf25dbda8715a27c4e49fe2f4096db54fb74e3002d23b WatchSource:0}: Error finding container 43efee53b6c8cc9e2eedf25dbda8715a27c4e49fe2f4096db54fb74e3002d23b: Status 404 returned error can't find the container with id 43efee53b6c8cc9e2eedf25dbda8715a27c4e49fe2f4096db54fb74e3002d23b
Oct 30 21:02:51.090498 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:51.090472    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-408918eb-f26js"
Oct 30 21:02:51.184485 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:51.183221    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d10d07e-a60e-4c6a-b067-66234939fa57-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d10d07e-a60e-4c6a-b067-66234939fa57-maas-api-openshift-ca\") pod \"7d10d07e-a60e-4c6a-b067-66234939fa57\" (UID: \"7d10d07e-a60e-4c6a-b067-66234939fa57\") "
Oct 30 21:02:51.184485 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:51.183444    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d10d07e-a60e-4c6a-b067-66234939fa57-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d10d07e-a60e-4c6a-b067-66234939fa57-maas-api-xks-ca\") pod \"7d10d07e-a60e-4c6a-b067-66234939fa57\" (UID: \"7d10d07e-a60e-4c6a-b067-66234939fa57\") "
Oct 30 21:02:51.184485 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:51.183561    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d10d07e-a60e-4c6a-b067-66234939fa57-kube-api-access-6znsz\" (UniqueName: \"kubernetes.io/projected/7d10d07e-a60e-4c6a-b067-66234939fa57-kube-api-access-6znsz\") pod \"7d10d07e-a60e-4c6a-b067-66234939fa57\" (UID: \"7d10d07e-a60e-4c6a-b067-66234939fa57\") "
Oct 30 21:02:51.187579 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:51.187549    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d10d07e-a60e-4c6a-b067-66234939fa57-maas-api-openshift-ca" pod "7d10d07e-a60e-4c6a-b067-66234939fa57" (UID: "7d10d07e-a60e-4c6a-b067-66234939fa57"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:02:51.196058 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:51.196010    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d10d07e-a60e-4c6a-b067-66234939fa57-maas-api-xks-ca" pod "7d10d07e-a60e-4c6a-b067-66234939fa57" (UID: "7d10d07e-a60e-4c6a-b067-66234939fa57"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:02:51.207110 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:51.207077    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d10d07e-a60e-4c6a-b067-66234939fa57-kube-api-access-6znsz" pod "7d10d07e-a60e-4c6a-b067-66234939fa57" (UID: "7d10d07e-a60e-4c6a-b067-66234939fa57"). InnerVolumeSpecName "kube-api-access-6znsz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:02:51.290347 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:51.287383    4521 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d10d07e-a60e-4c6a-b067-66234939fa57-maas-api-xks-ca\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:51.290347 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:51.287425    4521 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6znsz\" (UniqueName: \"kubernetes.io/projected/7d10d07e-a60e-4c6a-b067-66234939fa57-kube-api-access-6znsz\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:51.290347 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:51.287441    4521 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d10d07e-a60e-4c6a-b067-66234939fa57-maas-api-openshift-ca\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:51.895147 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:51.895118    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-408918eb-f26js"
Oct 30 21:02:51.895147 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:51.895131    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-408918eb-f26js" event={"ID":"7d10d07e-a60e-4c6a-b067-66234939fa57","Type":"ContainerDied","Data":"967aca5d2c6e3b1db0100f3d227c95655e0f7db62296606e1bb1560e12796b35"}
Oct 30 21:02:51.895572 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:51.895188    4521 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="967aca5d2c6e3b1db0100f3d227c95655e0f7db62296606e1bb1560e12796b35"
Oct 30 21:02:51.896745 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:51.896722    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nsppg" event={"ID":"47a667bb-4dac-484a-9f1b-01ef17c79710","Type":"ContainerStarted","Data":"43efee53b6c8cc9e2eedf25dbda8715a27c4e49fe2f4096db54fb74e3002d23b"}
Oct 30 21:02:51.898613 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:51.898591    4521 generic.go:356] "Generic (PLEG): container finished" podID="1e857e76-3d48-4e5b-a8b5-830cfac13f90" containerID="9e011a83e7dad15b724390446e366ea7ab9f3a5ba297591c56164b3b6ad78ab2" exitCode=0
Oct 30 21:02:51.898754 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:51.898658    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72ee85f3-2hq26" event={"ID":"1e857e76-3d48-4e5b-a8b5-830cfac13f90","Type":"ContainerDied","Data":"9e011a83e7dad15b724390446e366ea7ab9f3a5ba297591c56164b3b6ad78ab2"}
Oct 30 21:02:52.141313 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:52.141279    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-617df4d2-t4ck6"
Oct 30 21:02:52.198772 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:52.198727    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75d8f639-8cee-4be3-af28-eb7886627983-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75d8f639-8cee-4be3-af28-eb7886627983-maas-api-xks-ca\") pod \"75d8f639-8cee-4be3-af28-eb7886627983\" (UID: \"75d8f639-8cee-4be3-af28-eb7886627983\") "
Oct 30 21:02:52.198919 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:52.198847    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75d8f639-8cee-4be3-af28-eb7886627983-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75d8f639-8cee-4be3-af28-eb7886627983-maas-api-openshift-ca\") pod \"75d8f639-8cee-4be3-af28-eb7886627983\" (UID: \"75d8f639-8cee-4be3-af28-eb7886627983\") "
Oct 30 21:02:52.198972 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:52.198925    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75d8f639-8cee-4be3-af28-eb7886627983-kube-api-access-55bth\" (UniqueName: \"kubernetes.io/projected/75d8f639-8cee-4be3-af28-eb7886627983-kube-api-access-55bth\") pod \"75d8f639-8cee-4be3-af28-eb7886627983\" (UID: \"75d8f639-8cee-4be3-af28-eb7886627983\") "
Oct 30 21:02:52.200233 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:52.200202    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75d8f639-8cee-4be3-af28-eb7886627983-maas-api-openshift-ca" pod "75d8f639-8cee-4be3-af28-eb7886627983" (UID: "75d8f639-8cee-4be3-af28-eb7886627983"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:02:52.202204 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:52.202169    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75d8f639-8cee-4be3-af28-eb7886627983-kube-api-access-55bth" pod "75d8f639-8cee-4be3-af28-eb7886627983" (UID: "75d8f639-8cee-4be3-af28-eb7886627983"). InnerVolumeSpecName "kube-api-access-55bth". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:02:52.204395 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:52.204371    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75d8f639-8cee-4be3-af28-eb7886627983-maas-api-xks-ca" pod "75d8f639-8cee-4be3-af28-eb7886627983" (UID: "75d8f639-8cee-4be3-af28-eb7886627983"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:02:52.301009 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:52.300918    4521 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75d8f639-8cee-4be3-af28-eb7886627983-maas-api-xks-ca\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:52.301009 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:52.300960    4521 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75d8f639-8cee-4be3-af28-eb7886627983-maas-api-openshift-ca\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:52.301009 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:52.300977    4521 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55bth\" (UniqueName: \"kubernetes.io/projected/75d8f639-8cee-4be3-af28-eb7886627983-kube-api-access-55bth\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:52.923553 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:52.923518    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-617df4d2-t4ck6"
Oct 30 21:02:52.924121 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:52.924094    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-617df4d2-t4ck6" event={"ID":"75d8f639-8cee-4be3-af28-eb7886627983","Type":"ContainerDied","Data":"e86a3f1c53d6bb55b7d96466837556c6fb37e0f04a6b340098c84190f1d51014"}
Oct 30 21:02:52.924220 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:52.924138    4521 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e86a3f1c53d6bb55b7d96466837556c6fb37e0f04a6b340098c84190f1d51014"
Oct 30 21:02:53.164975 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:53.164658    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72ee85f3-2hq26"
Oct 30 21:02:53.235563 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:53.213843    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e857e76-3d48-4e5b-a8b5-830cfac13f90-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e857e76-3d48-4e5b-a8b5-830cfac13f90-maas-api-xks-ca\") pod \"1e857e76-3d48-4e5b-a8b5-830cfac13f90\" (UID: \"1e857e76-3d48-4e5b-a8b5-830cfac13f90\") "
Oct 30 21:02:53.235563 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:53.213893    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e857e76-3d48-4e5b-a8b5-830cfac13f90-kube-api-access-spnvs\" (UniqueName: \"kubernetes.io/projected/1e857e76-3d48-4e5b-a8b5-830cfac13f90-kube-api-access-spnvs\") pod \"1e857e76-3d48-4e5b-a8b5-830cfac13f90\" (UID: \"1e857e76-3d48-4e5b-a8b5-830cfac13f90\") "
Oct 30 21:02:53.235563 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:53.213966    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e857e76-3d48-4e5b-a8b5-830cfac13f90-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e857e76-3d48-4e5b-a8b5-830cfac13f90-maas-api-openshift-ca\") pod \"1e857e76-3d48-4e5b-a8b5-830cfac13f90\" (UID: \"1e857e76-3d48-4e5b-a8b5-830cfac13f90\") "
Oct 30 21:02:53.235563 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:53.215785    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e857e76-3d48-4e5b-a8b5-830cfac13f90-maas-api-openshift-ca" pod "1e857e76-3d48-4e5b-a8b5-830cfac13f90" (UID: "1e857e76-3d48-4e5b-a8b5-830cfac13f90"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:02:53.284941 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:53.284816    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e857e76-3d48-4e5b-a8b5-830cfac13f90-maas-api-xks-ca" pod "1e857e76-3d48-4e5b-a8b5-830cfac13f90" (UID: "1e857e76-3d48-4e5b-a8b5-830cfac13f90"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:02:53.309549 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:53.309500    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e857e76-3d48-4e5b-a8b5-830cfac13f90-kube-api-access-spnvs" pod "1e857e76-3d48-4e5b-a8b5-830cfac13f90" (UID: "1e857e76-3d48-4e5b-a8b5-830cfac13f90"). InnerVolumeSpecName "kube-api-access-spnvs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:02:53.315630 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:53.315566    4521 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-spnvs\" (UniqueName: \"kubernetes.io/projected/1e857e76-3d48-4e5b-a8b5-830cfac13f90-kube-api-access-spnvs\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:53.315630 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:53.315590    4521 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e857e76-3d48-4e5b-a8b5-830cfac13f90-maas-api-openshift-ca\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:53.315630 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:53.315602    4521 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e857e76-3d48-4e5b-a8b5-830cfac13f90-maas-api-xks-ca\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:53.699004 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:53.698972    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-39fb96-openshift-default-5f55489475-z6jxt"
Oct 30 21:02:53.824207 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:53.824173    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3990d1c-906f-4267-b67b-5b1309f20cfc-istio-token\" (UniqueName: \"kubernetes.io/projected/d3990d1c-906f-4267-b67b-5b1309f20cfc-istio-token\") pod \"d3990d1c-906f-4267-b67b-5b1309f20cfc\" (UID: \"d3990d1c-906f-4267-b67b-5b1309f20cfc\") "
Oct 30 21:02:53.824411 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:53.824282    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3990d1c-906f-4267-b67b-5b1309f20cfc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3990d1c-906f-4267-b67b-5b1309f20cfc-credential-socket\") pod \"d3990d1c-906f-4267-b67b-5b1309f20cfc\" (UID: \"d3990d1c-906f-4267-b67b-5b1309f20cfc\") "
Oct 30 21:02:53.824737 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:53.824666    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3990d1c-906f-4267-b67b-5b1309f20cfc-credential-socket" pod "d3990d1c-906f-4267-b67b-5b1309f20cfc" (UID: "d3990d1c-906f-4267-b67b-5b1309f20cfc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:02:53.824870 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:53.824741    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3990d1c-906f-4267-b67b-5b1309f20cfc-workload-certs" pod "d3990d1c-906f-4267-b67b-5b1309f20cfc" (UID: "d3990d1c-906f-4267-b67b-5b1309f20cfc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:02:53.824870 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:53.824762    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3990d1c-906f-4267-b67b-5b1309f20cfc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3990d1c-906f-4267-b67b-5b1309f20cfc-workload-certs\") pod \"d3990d1c-906f-4267-b67b-5b1309f20cfc\" (UID: \"d3990d1c-906f-4267-b67b-5b1309f20cfc\") "
Oct 30 21:02:53.824870 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:53.824836    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3990d1c-906f-4267-b67b-5b1309f20cfc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3990d1c-906f-4267-b67b-5b1309f20cfc-istio-data\") pod \"d3990d1c-906f-4267-b67b-5b1309f20cfc\" (UID: \"d3990d1c-906f-4267-b67b-5b1309f20cfc\") "
Oct 30 21:02:53.825100 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:53.825062    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3990d1c-906f-4267-b67b-5b1309f20cfc-kube-api-access-7dfzh\" (UniqueName: \"kubernetes.io/projected/d3990d1c-906f-4267-b67b-5b1309f20cfc-kube-api-access-7dfzh\") pod \"d3990d1c-906f-4267-b67b-5b1309f20cfc\" (UID: \"d3990d1c-906f-4267-b67b-5b1309f20cfc\") "
Oct 30 21:02:53.825172 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:53.825152    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d3990d1c-906f-4267-b67b-5b1309f20cfc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3990d1c-906f-4267-b67b-5b1309f20cfc-istio-podinfo\") pod \"d3990d1c-906f-4267-b67b-5b1309f20cfc\" (UID: \"d3990d1c-906f-4267-b67b-5b1309f20cfc\") "
Oct 30 21:02:53.825392 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:53.825371    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3990d1c-906f-4267-b67b-5b1309f20cfc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3990d1c-906f-4267-b67b-5b1309f20cfc-istio-envoy\") pod \"d3990d1c-906f-4267-b67b-5b1309f20cfc\" (UID: \"d3990d1c-906f-4267-b67b-5b1309f20cfc\") "
Oct 30 21:02:53.825466 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:53.825413    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3990d1c-906f-4267-b67b-5b1309f20cfc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3990d1c-906f-4267-b67b-5b1309f20cfc-istiod-ca-cert\") pod \"d3990d1c-906f-4267-b67b-5b1309f20cfc\" (UID: \"d3990d1c-906f-4267-b67b-5b1309f20cfc\") "
Oct 30 21:02:53.825517 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:53.825497    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3990d1c-906f-4267-b67b-5b1309f20cfc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3990d1c-906f-4267-b67b-5b1309f20cfc-workload-socket\") pod \"d3990d1c-906f-4267-b67b-5b1309f20cfc\" (UID: \"d3990d1c-906f-4267-b67b-5b1309f20cfc\") "
Oct 30 21:02:53.826020 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:53.825997    4521 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3990d1c-906f-4267-b67b-5b1309f20cfc-credential-socket\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:53.826104 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:53.826025    4521 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3990d1c-906f-4267-b67b-5b1309f20cfc-workload-certs\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:53.826253 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:53.826234    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3990d1c-906f-4267-b67b-5b1309f20cfc-workload-socket" pod "d3990d1c-906f-4267-b67b-5b1309f20cfc" (UID: "d3990d1c-906f-4267-b67b-5b1309f20cfc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:02:53.826785 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:53.826747    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3990d1c-906f-4267-b67b-5b1309f20cfc-istio-data" pod "d3990d1c-906f-4267-b67b-5b1309f20cfc" (UID: "d3990d1c-906f-4267-b67b-5b1309f20cfc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:02:53.826885 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:53.826862    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3990d1c-906f-4267-b67b-5b1309f20cfc-istiod-ca-cert" pod "d3990d1c-906f-4267-b67b-5b1309f20cfc" (UID: "d3990d1c-906f-4267-b67b-5b1309f20cfc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:02:53.828792 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:53.828771    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3990d1c-906f-4267-b67b-5b1309f20cfc-istio-envoy" pod "d3990d1c-906f-4267-b67b-5b1309f20cfc" (UID: "d3990d1c-906f-4267-b67b-5b1309f20cfc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:02:53.828892 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:53.828875    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3990d1c-906f-4267-b67b-5b1309f20cfc-kube-api-access-7dfzh" pod "d3990d1c-906f-4267-b67b-5b1309f20cfc" (UID: "d3990d1c-906f-4267-b67b-5b1309f20cfc"). InnerVolumeSpecName "kube-api-access-7dfzh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:02:53.830851 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:53.830823    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3990d1c-906f-4267-b67b-5b1309f20cfc-istio-token" pod "d3990d1c-906f-4267-b67b-5b1309f20cfc" (UID: "d3990d1c-906f-4267-b67b-5b1309f20cfc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:02:53.832114 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:53.832087    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d3990d1c-906f-4267-b67b-5b1309f20cfc-istio-podinfo" pod "d3990d1c-906f-4267-b67b-5b1309f20cfc" (UID: "d3990d1c-906f-4267-b67b-5b1309f20cfc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:02:53.930183 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:53.929422    4521 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3990d1c-906f-4267-b67b-5b1309f20cfc-istio-envoy\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:53.930183 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:53.929462    4521 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3990d1c-906f-4267-b67b-5b1309f20cfc-istiod-ca-cert\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:53.930183 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:53.929476    4521 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3990d1c-906f-4267-b67b-5b1309f20cfc-workload-socket\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:53.930183 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:53.929488    4521 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3990d1c-906f-4267-b67b-5b1309f20cfc-istio-token\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:53.930183 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:53.929500    4521 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3990d1c-906f-4267-b67b-5b1309f20cfc-istio-data\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:53.930183 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:53.929513    4521 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7dfzh\" (UniqueName: \"kubernetes.io/projected/d3990d1c-906f-4267-b67b-5b1309f20cfc-kube-api-access-7dfzh\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:53.930183 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:53.929526    4521 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3990d1c-906f-4267-b67b-5b1309f20cfc-istio-podinfo\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:53.951353 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:53.941472    4521 generic.go:356] "Generic (PLEG): container finished" podID="d3990d1c-906f-4267-b67b-5b1309f20cfc" containerID="d2435b724e4f5fc6ce200c74e76d64b9c4188f66739b19ce5da0d0800f4797fa" exitCode=0
Oct 30 21:02:53.951353 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:53.941546    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-39fb96-openshift-default-5f55489475-z6jxt"
Oct 30 21:02:53.951353 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:53.941659    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-39fb96-openshift-default-5f55489475-z6jxt" event={"ID":"d3990d1c-906f-4267-b67b-5b1309f20cfc","Type":"ContainerDied","Data":"d2435b724e4f5fc6ce200c74e76d64b9c4188f66739b19ce5da0d0800f4797fa"}
Oct 30 21:02:53.951353 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:53.941698    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-39fb96-openshift-default-5f55489475-z6jxt" event={"ID":"d3990d1c-906f-4267-b67b-5b1309f20cfc","Type":"ContainerDied","Data":"9a0fda3a0c45c18c6503f7bc0706c18db8cae2806bea478ba2d0875c3386ea5c"}
Oct 30 21:02:53.951353 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:53.941727    4521 scope.go:122] "RemoveContainer" containerID="d2435b724e4f5fc6ce200c74e76d64b9c4188f66739b19ce5da0d0800f4797fa"
Oct 30 21:02:53.951353 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:53.949774    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72ee85f3-2hq26"
Oct 30 21:02:53.951353 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:53.949775    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72ee85f3-2hq26" event={"ID":"1e857e76-3d48-4e5b-a8b5-830cfac13f90","Type":"ContainerDied","Data":"36e13db3282053822160ff0d396939c7cf6fe65571880e8d18973180a449615e"}
Oct 30 21:02:53.951353 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:53.949903    4521 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="36e13db3282053822160ff0d396939c7cf6fe65571880e8d18973180a449615e"
Oct 30 21:02:53.970193 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:53.970165    4521 scope.go:122] "RemoveContainer" containerID="d2435b724e4f5fc6ce200c74e76d64b9c4188f66739b19ce5da0d0800f4797fa"
Oct 30 21:02:53.970837 ip-10-0-131-125 kubenswrapper[4521]: E1030 21:02:53.970803    4521 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2435b724e4f5fc6ce200c74e76d64b9c4188f66739b19ce5da0d0800f4797fa\": container with ID starting with d2435b724e4f5fc6ce200c74e76d64b9c4188f66739b19ce5da0d0800f4797fa not found: ID does not exist" containerID="d2435b724e4f5fc6ce200c74e76d64b9c4188f66739b19ce5da0d0800f4797fa"
Oct 30 21:02:53.970945 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:53.970845    4521 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2435b724e4f5fc6ce200c74e76d64b9c4188f66739b19ce5da0d0800f4797fa"} err="failed to get container status \"d2435b724e4f5fc6ce200c74e76d64b9c4188f66739b19ce5da0d0800f4797fa\": rpc error: code = NotFound desc = could not find container \"d2435b724e4f5fc6ce200c74e76d64b9c4188f66739b19ce5da0d0800f4797fa\": container with ID starting with d2435b724e4f5fc6ce200c74e76d64b9c4188f66739b19ce5da0d0800f4797fa not found: ID does not exist"
Oct 30 21:02:53.975581 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:53.974234    4521 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-39fb96-openshift-default-5f55489475-z6jxt"]
Oct 30 21:02:53.980956 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:53.980927    4521 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-39fb96-openshift-default-5f55489475-z6jxt"]
Oct 30 21:02:54.577251 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:54.577212    4521 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:02:54.577382 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:54.577302    4521 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:02:54.837631 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:54.834196    4521 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3990d1c-906f-4267-b67b-5b1309f20cfc" path="/var/lib/kubelet/pods/d3990d1c-906f-4267-b67b-5b1309f20cfc/volumes"
Oct 30 21:02:54.842655 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:54.840121    4521 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-hlv6g"]
Oct 30 21:02:54.856344 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:54.852707    4521 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-hlv6g" podUID="f860c372-4493-4c25-b46a-a6d3af710390" containerName="istio-proxy" containerID="cri-o://21a3de3b202c4f10c9229e21a63d3e7b1b879d9996e96d4490513f2692dea48a" gracePeriod=30
Oct 30 21:02:54.869355 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:54.867147    4521 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-jjmw4"]
Oct 30 21:02:54.869355 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:54.867409    4521 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-jjmw4" podUID="d91a4f29-950e-4370-92e0-8ec493b72620" containerName="istio-proxy" containerID="cri-o://949c736f77bbe18f449157304ec69d5f8dc30d34424b7bec3a13bef4c4ece749" gracePeriod=30
Oct 30 21:02:54.870580 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:54.870552    4521 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-5tqcb"]
Oct 30 21:02:54.871058 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:54.870999    4521 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-5tqcb" podUID="11b85d0f-fa82-4861-9c1c-e47a1c1c308a" containerName="istio-proxy" containerID="cri-o://b251af91f4b3645d981cdae3ca14049d5f598b6d3c79722fa36f70e80e806ab9" gracePeriod=30
Oct 30 21:02:55.014467 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:55.014423    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nsppg" event={"ID":"47a667bb-4dac-484a-9f1b-01ef17c79710","Type":"ContainerStarted","Data":"c7c86af13ef1f3f770dbe47e5510e2f31901d0a3ea8c310e29023f3024a816a4"}
Oct 30 21:02:55.022346 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:55.016577    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nsppg"
Oct 30 21:02:55.044860 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:55.037567    4521 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-nsppg container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 21:02:55.045127 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:55.045102    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nsppg" podUID="47a667bb-4dac-484a-9f1b-01ef17c79710" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 21:02:55.557688 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:55.557615    4521 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nsppg" podStartSLOduration=2.014381349 podStartE2EDuration="5.557592322s" totalImagesPullingTime="3.543210973s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:02:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:02:55.085536427 +0000 UTC m=+1442.431157668" watchObservedRunningTime="2026-10-30 21:02:55.557592322 +0000 UTC m=+1442.903213562"
Oct 30 21:02:55.561682 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:55.561644    4521 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-65968b-6876d4dbfb-js2xm"]
Oct 30 21:02:55.561912 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:55.561883    4521 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-65968b-6876d4dbfb-js2xm" podUID="7cb73975-c1ac-4957-9337-6ada4d05a2dc" containerName="maas-api" containerID="cri-o://5fc3147ddba94cc576ec94aa37ad1ff19456d41f1bcff9c22589c1333e26c581" gracePeriod=30
Oct 30 21:02:55.811438 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:55.811340    4521 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-65968b-hfxjz"]
Oct 30 21:02:55.824458 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:55.824427    4521 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-65968b-hfxjz"]
Oct 30 21:02:56.117388 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:56.117238    4521 generic.go:356] "Generic (PLEG): container finished" podID="7cb73975-c1ac-4957-9337-6ada4d05a2dc" containerID="5fc3147ddba94cc576ec94aa37ad1ff19456d41f1bcff9c22589c1333e26c581" exitCode=0
Oct 30 21:02:56.119539 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:56.119510    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-65968b-6876d4dbfb-js2xm" event={"ID":"7cb73975-c1ac-4957-9337-6ada4d05a2dc","Type":"ContainerDied","Data":"5fc3147ddba94cc576ec94aa37ad1ff19456d41f1bcff9c22589c1333e26c581"}
Oct 30 21:02:56.120169 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:56.120152    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nsppg"
Oct 30 21:02:56.182691 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:56.182662    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-65968b-6876d4dbfb-js2xm"
Oct 30 21:02:56.304854 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:56.304810    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7cb73975-c1ac-4957-9337-6ada4d05a2dc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7cb73975-c1ac-4957-9337-6ada4d05a2dc-metrics-certs\") pod \"7cb73975-c1ac-4957-9337-6ada4d05a2dc\" (UID: \"7cb73975-c1ac-4957-9337-6ada4d05a2dc\") "
Oct 30 21:02:56.305036 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:56.304928    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7cb73975-c1ac-4957-9337-6ada4d05a2dc-kube-api-access-56bp7\" (UniqueName: \"kubernetes.io/projected/7cb73975-c1ac-4957-9337-6ada4d05a2dc-kube-api-access-56bp7\") pod \"7cb73975-c1ac-4957-9337-6ada4d05a2dc\" (UID: \"7cb73975-c1ac-4957-9337-6ada4d05a2dc\") "
Oct 30 21:02:56.305095 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:56.305035    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7cb73975-c1ac-4957-9337-6ada4d05a2dc-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cb73975-c1ac-4957-9337-6ada4d05a2dc-openshift-service-ca\") pod \"7cb73975-c1ac-4957-9337-6ada4d05a2dc\" (UID: \"7cb73975-c1ac-4957-9337-6ada4d05a2dc\") "
Oct 30 21:02:56.305182 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:56.305148    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7cb73975-c1ac-4957-9337-6ada4d05a2dc-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7cb73975-c1ac-4957-9337-6ada4d05a2dc-maas-api-tls\") pod \"7cb73975-c1ac-4957-9337-6ada4d05a2dc\" (UID: \"7cb73975-c1ac-4957-9337-6ada4d05a2dc\") "
Oct 30 21:02:56.305524 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:56.305499    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cb73975-c1ac-4957-9337-6ada4d05a2dc-openshift-service-ca" pod "7cb73975-c1ac-4957-9337-6ada4d05a2dc" (UID: "7cb73975-c1ac-4957-9337-6ada4d05a2dc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:02:56.305821 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:56.305794    4521 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cb73975-c1ac-4957-9337-6ada4d05a2dc-openshift-service-ca\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:56.307261 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:56.307235    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7cb73975-c1ac-4957-9337-6ada4d05a2dc-metrics-certs" pod "7cb73975-c1ac-4957-9337-6ada4d05a2dc" (UID: "7cb73975-c1ac-4957-9337-6ada4d05a2dc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:02:56.307483 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:56.307264    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7cb73975-c1ac-4957-9337-6ada4d05a2dc-maas-api-tls" pod "7cb73975-c1ac-4957-9337-6ada4d05a2dc" (UID: "7cb73975-c1ac-4957-9337-6ada4d05a2dc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:02:56.307483 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:56.307347    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cb73975-c1ac-4957-9337-6ada4d05a2dc-kube-api-access-56bp7" pod "7cb73975-c1ac-4957-9337-6ada4d05a2dc" (UID: "7cb73975-c1ac-4957-9337-6ada4d05a2dc"). InnerVolumeSpecName "kube-api-access-56bp7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:02:56.407426 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:56.407307    4521 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7cb73975-c1ac-4957-9337-6ada4d05a2dc-maas-api-tls\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:56.407426 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:56.407376    4521 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7cb73975-c1ac-4957-9337-6ada4d05a2dc-metrics-certs\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:56.407426 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:56.407394    4521 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56bp7\" (UniqueName: \"kubernetes.io/projected/7cb73975-c1ac-4957-9337-6ada4d05a2dc-kube-api-access-56bp7\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:56.420609 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:56.420567    4521 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c2d1eb-9m5xb"]
Oct 30 21:02:56.424934 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:56.424899    4521 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c2d1eb-9m5xb"]
Oct 30 21:02:56.792902 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:56.792858    4521 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1078ebe9-650f-4a2c-9347-dee3fd88e4de" path="/var/lib/kubelet/pods/1078ebe9-650f-4a2c-9347-dee3fd88e4de/volumes"
Oct 30 21:02:56.793512 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:56.793494    4521 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28e05ac8-5b4d-4085-bf8c-58793ca1c370" path="/var/lib/kubelet/pods/28e05ac8-5b4d-4085-bf8c-58793ca1c370/volumes"
Oct 30 21:02:57.128421 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:57.128317    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-65968b-6876d4dbfb-js2xm"
Oct 30 21:02:57.128421 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:57.128350    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-65968b-6876d4dbfb-js2xm" event={"ID":"7cb73975-c1ac-4957-9337-6ada4d05a2dc","Type":"ContainerDied","Data":"52b6b7ba03575592d77d6ea5013927bf2423f6c41d112f30cfbe3396a9732293"}
Oct 30 21:02:57.128421 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:57.128422    4521 scope.go:122] "RemoveContainer" containerID="5fc3147ddba94cc576ec94aa37ad1ff19456d41f1bcff9c22589c1333e26c581"
Oct 30 21:02:57.155008 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:57.154974    4521 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-65968b-6876d4dbfb-js2xm"]
Oct 30 21:02:57.159749 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:57.159719    4521 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-65968b-6876d4dbfb-js2xm"]
Oct 30 21:02:58.793410 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:02:58.793377    4521 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7cb73975-c1ac-4957-9337-6ada4d05a2dc" path="/var/lib/kubelet/pods/7cb73975-c1ac-4957-9337-6ada4d05a2dc/volumes"
Oct 30 21:03:00.168636 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:00.168599    4521 generic.go:356] "Generic (PLEG): container finished" podID="f860c372-4493-4c25-b46a-a6d3af710390" containerID="21a3de3b202c4f10c9229e21a63d3e7b1b879d9996e96d4490513f2692dea48a" exitCode=0
Oct 30 21:03:00.169056 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:00.168692    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-hlv6g" event={"ID":"f860c372-4493-4c25-b46a-a6d3af710390","Type":"ContainerDied","Data":"21a3de3b202c4f10c9229e21a63d3e7b1b879d9996e96d4490513f2692dea48a"}
Oct 30 21:03:00.169056 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:00.168783    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-hlv6g" event={"ID":"f860c372-4493-4c25-b46a-a6d3af710390","Type":"ContainerDied","Data":"d08a5ea3aae2a3215a9800e6f9200b7bb7f399b060db228a11c439e59390bd63"}
Oct 30 21:03:00.169056 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:00.168809    4521 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d08a5ea3aae2a3215a9800e6f9200b7bb7f399b060db228a11c439e59390bd63"
Oct 30 21:03:00.170992 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:00.170961    4521 generic.go:356] "Generic (PLEG): container finished" podID="11b85d0f-fa82-4861-9c1c-e47a1c1c308a" containerID="b251af91f4b3645d981cdae3ca14049d5f598b6d3c79722fa36f70e80e806ab9" exitCode=0
Oct 30 21:03:00.171121 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:00.171034    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-5tqcb" event={"ID":"11b85d0f-fa82-4861-9c1c-e47a1c1c308a","Type":"ContainerDied","Data":"b251af91f4b3645d981cdae3ca14049d5f598b6d3c79722fa36f70e80e806ab9"}
Oct 30 21:03:00.173489 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:00.173379    4521 generic.go:356] "Generic (PLEG): container finished" podID="d91a4f29-950e-4370-92e0-8ec493b72620" containerID="949c736f77bbe18f449157304ec69d5f8dc30d34424b7bec3a13bef4c4ece749" exitCode=0
Oct 30 21:03:00.173608 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:00.173510    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-jjmw4" event={"ID":"d91a4f29-950e-4370-92e0-8ec493b72620","Type":"ContainerDied","Data":"949c736f77bbe18f449157304ec69d5f8dc30d34424b7bec3a13bef4c4ece749"}
Oct 30 21:03:00.175832 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:00.175814    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-hlv6g"
Oct 30 21:03:00.179703 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:00.179678    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f860c372-4493-4c25-b46a-a6d3af710390-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f860c372-4493-4c25-b46a-a6d3af710390-istio-data\") pod \"f860c372-4493-4c25-b46a-a6d3af710390\" (UID: \"f860c372-4493-4c25-b46a-a6d3af710390\") "
Oct 30 21:03:00.179936 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:00.179751    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f860c372-4493-4c25-b46a-a6d3af710390-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f860c372-4493-4c25-b46a-a6d3af710390-workload-socket\") pod \"f860c372-4493-4c25-b46a-a6d3af710390\" (UID: \"f860c372-4493-4c25-b46a-a6d3af710390\") "
Oct 30 21:03:00.179936 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:00.179801    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f860c372-4493-4c25-b46a-a6d3af710390-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f860c372-4493-4c25-b46a-a6d3af710390-credential-socket\") pod \"f860c372-4493-4c25-b46a-a6d3af710390\" (UID: \"f860c372-4493-4c25-b46a-a6d3af710390\") "
Oct 30 21:03:00.179936 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:00.179871    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f860c372-4493-4c25-b46a-a6d3af710390-kube-api-access-kp9px\" (UniqueName: \"kubernetes.io/projected/f860c372-4493-4c25-b46a-a6d3af710390-kube-api-access-kp9px\") pod \"f860c372-4493-4c25-b46a-a6d3af710390\" (UID: \"f860c372-4493-4c25-b46a-a6d3af710390\") "
Oct 30 21:03:00.182037 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:00.180439    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f860c372-4493-4c25-b46a-a6d3af710390-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f860c372-4493-4c25-b46a-a6d3af710390-workload-certs\") pod \"f860c372-4493-4c25-b46a-a6d3af710390\" (UID: \"f860c372-4493-4c25-b46a-a6d3af710390\") "
Oct 30 21:03:00.182037 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:00.180507    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f860c372-4493-4c25-b46a-a6d3af710390-istio-token\" (UniqueName: \"kubernetes.io/projected/f860c372-4493-4c25-b46a-a6d3af710390-istio-token\") pod \"f860c372-4493-4c25-b46a-a6d3af710390\" (UID: \"f860c372-4493-4c25-b46a-a6d3af710390\") "
Oct 30 21:03:00.182037 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:00.180536    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f860c372-4493-4c25-b46a-a6d3af710390-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f860c372-4493-4c25-b46a-a6d3af710390-istio-envoy\") pod \"f860c372-4493-4c25-b46a-a6d3af710390\" (UID: \"f860c372-4493-4c25-b46a-a6d3af710390\") "
Oct 30 21:03:00.182037 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:00.180625    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f860c372-4493-4c25-b46a-a6d3af710390-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f860c372-4493-4c25-b46a-a6d3af710390-istiod-ca-cert\") pod \"f860c372-4493-4c25-b46a-a6d3af710390\" (UID: \"f860c372-4493-4c25-b46a-a6d3af710390\") "
Oct 30 21:03:00.182037 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:00.180775    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f860c372-4493-4c25-b46a-a6d3af710390-credential-socket" pod "f860c372-4493-4c25-b46a-a6d3af710390" (UID: "f860c372-4493-4c25-b46a-a6d3af710390"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:03:00.182037 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:00.180810    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f860c372-4493-4c25-b46a-a6d3af710390-workload-certs" pod "f860c372-4493-4c25-b46a-a6d3af710390" (UID: "f860c372-4493-4c25-b46a-a6d3af710390"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:03:00.182037 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:00.181396    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f860c372-4493-4c25-b46a-a6d3af710390-istiod-ca-cert" pod "f860c372-4493-4c25-b46a-a6d3af710390" (UID: "f860c372-4493-4c25-b46a-a6d3af710390"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:03:00.182037 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:00.181452    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f860c372-4493-4c25-b46a-a6d3af710390-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f860c372-4493-4c25-b46a-a6d3af710390-istio-podinfo\") pod \"f860c372-4493-4c25-b46a-a6d3af710390\" (UID: \"f860c372-4493-4c25-b46a-a6d3af710390\") "
Oct 30 21:03:00.182037 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:00.181987    4521 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f860c372-4493-4c25-b46a-a6d3af710390-credential-socket\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:03:00.182037 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:00.182005    4521 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f860c372-4493-4c25-b46a-a6d3af710390-workload-certs\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:03:00.182037 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:00.182017    4521 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f860c372-4493-4c25-b46a-a6d3af710390-istiod-ca-cert\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:03:00.183268 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:00.183239    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f860c372-4493-4c25-b46a-a6d3af710390-istio-envoy" pod "f860c372-4493-4c25-b46a-a6d3af710390" (UID: "f860c372-4493-4c25-b46a-a6d3af710390"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:03:00.184160 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:00.184113    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f860c372-4493-4c25-b46a-a6d3af710390-istio-data" pod "f860c372-4493-4c25-b46a-a6d3af710390" (UID: "f860c372-4493-4c25-b46a-a6d3af710390"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:03:00.184598 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:00.184575    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f860c372-4493-4c25-b46a-a6d3af710390-workload-socket" pod "f860c372-4493-4c25-b46a-a6d3af710390" (UID: "f860c372-4493-4c25-b46a-a6d3af710390"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:03:00.186207 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:00.186122    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f860c372-4493-4c25-b46a-a6d3af710390-istio-token" pod "f860c372-4493-4c25-b46a-a6d3af710390" (UID: "f860c372-4493-4c25-b46a-a6d3af710390"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:03:00.190009 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:00.189708    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f860c372-4493-4c25-b46a-a6d3af710390-istio-podinfo" pod "f860c372-4493-4c25-b46a-a6d3af710390" (UID: "f860c372-4493-4c25-b46a-a6d3af710390"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:03:00.190009 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:00.189797    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f860c372-4493-4c25-b46a-a6d3af710390-kube-api-access-kp9px" pod "f860c372-4493-4c25-b46a-a6d3af710390" (UID: "f860c372-4493-4c25-b46a-a6d3af710390"). InnerVolumeSpecName "kube-api-access-kp9px". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:03:00.207091 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:00.207059    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-jjmw4"
Oct 30 21:03:00.252411 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:00.252061    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-5tqcb"
Oct 30 21:03:00.283427 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:00.283396    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d91a4f29-950e-4370-92e0-8ec493b72620-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d91a4f29-950e-4370-92e0-8ec493b72620-istio-envoy\") pod \"d91a4f29-950e-4370-92e0-8ec493b72620\" (UID: \"d91a4f29-950e-4370-92e0-8ec493b72620\") "
Oct 30 21:03:00.283656 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:00.283472    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11b85d0f-fa82-4861-9c1c-e47a1c1c308a-kube-api-access-m6lrz\" (UniqueName: \"kubernetes.io/projected/11b85d0f-fa82-4861-9c1c-e47a1c1c308a-kube-api-access-m6lrz\") pod \"11b85d0f-fa82-4861-9c1c-e47a1c1c308a\" (UID: \"11b85d0f-fa82-4861-9c1c-e47a1c1c308a\") "
Oct 30 21:03:00.283656 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:00.283517    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11b85d0f-fa82-4861-9c1c-e47a1c1c308a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11b85d0f-fa82-4861-9c1c-e47a1c1c308a-credential-socket\") pod \"11b85d0f-fa82-4861-9c1c-e47a1c1c308a\" (UID: \"11b85d0f-fa82-4861-9c1c-e47a1c1c308a\") "
Oct 30 21:03:00.283656 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:00.283553    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d91a4f29-950e-4370-92e0-8ec493b72620-istio-token\" (UniqueName: \"kubernetes.io/projected/d91a4f29-950e-4370-92e0-8ec493b72620-istio-token\") pod \"d91a4f29-950e-4370-92e0-8ec493b72620\" (UID: \"d91a4f29-950e-4370-92e0-8ec493b72620\") "
Oct 30 21:03:00.283656 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:00.283590    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d91a4f29-950e-4370-92e0-8ec493b72620-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d91a4f29-950e-4370-92e0-8ec493b72620-istio-podinfo\") pod \"d91a4f29-950e-4370-92e0-8ec493b72620\" (UID: \"d91a4f29-950e-4370-92e0-8ec493b72620\") "
Oct 30 21:03:00.283656 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:00.283627    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/11b85d0f-fa82-4861-9c1c-e47a1c1c308a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11b85d0f-fa82-4861-9c1c-e47a1c1c308a-istio-podinfo\") pod \"11b85d0f-fa82-4861-9c1c-e47a1c1c308a\" (UID: \"11b85d0f-fa82-4861-9c1c-e47a1c1c308a\") "
Oct 30 21:03:00.284023 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:00.283662    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d91a4f29-950e-4370-92e0-8ec493b72620-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d91a4f29-950e-4370-92e0-8ec493b72620-istio-data\") pod \"d91a4f29-950e-4370-92e0-8ec493b72620\" (UID: \"d91a4f29-950e-4370-92e0-8ec493b72620\") "
Oct 30 21:03:00.284023 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:00.283740    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d91a4f29-950e-4370-92e0-8ec493b72620-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d91a4f29-950e-4370-92e0-8ec493b72620-istiod-ca-cert\") pod \"d91a4f29-950e-4370-92e0-8ec493b72620\" (UID: \"d91a4f29-950e-4370-92e0-8ec493b72620\") "
Oct 30 21:03:00.284023 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:00.283780    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11b85d0f-fa82-4861-9c1c-e47a1c1c308a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11b85d0f-fa82-4861-9c1c-e47a1c1c308a-istio-data\") pod \"11b85d0f-fa82-4861-9c1c-e47a1c1c308a\" (UID: \"11b85d0f-fa82-4861-9c1c-e47a1c1c308a\") "
Oct 30 21:03:00.284023 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:00.283829    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d91a4f29-950e-4370-92e0-8ec493b72620-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d91a4f29-950e-4370-92e0-8ec493b72620-credential-socket\") pod \"d91a4f29-950e-4370-92e0-8ec493b72620\" (UID: \"d91a4f29-950e-4370-92e0-8ec493b72620\") "
Oct 30 21:03:00.284023 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:00.283859    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11b85d0f-fa82-4861-9c1c-e47a1c1c308a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11b85d0f-fa82-4861-9c1c-e47a1c1c308a-workload-certs\") pod \"11b85d0f-fa82-4861-9c1c-e47a1c1c308a\" (UID: \"11b85d0f-fa82-4861-9c1c-e47a1c1c308a\") "
Oct 30 21:03:00.284023 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:00.283908    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d91a4f29-950e-4370-92e0-8ec493b72620-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d91a4f29-950e-4370-92e0-8ec493b72620-workload-certs\") pod \"d91a4f29-950e-4370-92e0-8ec493b72620\" (UID: \"d91a4f29-950e-4370-92e0-8ec493b72620\") "
Oct 30 21:03:00.284023 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:00.283934    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11b85d0f-fa82-4861-9c1c-e47a1c1c308a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11b85d0f-fa82-4861-9c1c-e47a1c1c308a-istiod-ca-cert\") pod \"11b85d0f-fa82-4861-9c1c-e47a1c1c308a\" (UID: \"11b85d0f-fa82-4861-9c1c-e47a1c1c308a\") "
Oct 30 21:03:00.284023 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:00.284017    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11b85d0f-fa82-4861-9c1c-e47a1c1c308a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11b85d0f-fa82-4861-9c1c-e47a1c1c308a-istio-envoy\") pod \"11b85d0f-fa82-4861-9c1c-e47a1c1c308a\" (UID: \"11b85d0f-fa82-4861-9c1c-e47a1c1c308a\") "
Oct 30 21:03:00.284549 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:00.284050    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11b85d0f-fa82-4861-9c1c-e47a1c1c308a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11b85d0f-fa82-4861-9c1c-e47a1c1c308a-workload-socket\") pod \"11b85d0f-fa82-4861-9c1c-e47a1c1c308a\" (UID: \"11b85d0f-fa82-4861-9c1c-e47a1c1c308a\") "
Oct 30 21:03:00.284549 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:00.284097    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d91a4f29-950e-4370-92e0-8ec493b72620-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d91a4f29-950e-4370-92e0-8ec493b72620-workload-socket\") pod \"d91a4f29-950e-4370-92e0-8ec493b72620\" (UID: \"d91a4f29-950e-4370-92e0-8ec493b72620\") "
Oct 30 21:03:00.284549 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:00.284130    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11b85d0f-fa82-4861-9c1c-e47a1c1c308a-istio-token\" (UniqueName: \"kubernetes.io/projected/11b85d0f-fa82-4861-9c1c-e47a1c1c308a-istio-token\") pod \"11b85d0f-fa82-4861-9c1c-e47a1c1c308a\" (UID: \"11b85d0f-fa82-4861-9c1c-e47a1c1c308a\") "
Oct 30 21:03:00.284549 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:00.284211    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d91a4f29-950e-4370-92e0-8ec493b72620-kube-api-access-vqs62\" (UniqueName: \"kubernetes.io/projected/d91a4f29-950e-4370-92e0-8ec493b72620-kube-api-access-vqs62\") pod \"d91a4f29-950e-4370-92e0-8ec493b72620\" (UID: \"d91a4f29-950e-4370-92e0-8ec493b72620\") "
Oct 30 21:03:00.284802 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:00.284739    4521 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kp9px\" (UniqueName: \"kubernetes.io/projected/f860c372-4493-4c25-b46a-a6d3af710390-kube-api-access-kp9px\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:03:00.284802 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:00.284765    4521 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f860c372-4493-4c25-b46a-a6d3af710390-istio-token\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:03:00.284802 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:00.284781    4521 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f860c372-4493-4c25-b46a-a6d3af710390-istio-envoy\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:03:00.284802 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:00.284794    4521 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f860c372-4493-4c25-b46a-a6d3af710390-istio-podinfo\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:03:00.284802 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:00.284807    4521 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f860c372-4493-4c25-b46a-a6d3af710390-istio-data\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:03:00.285543 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:00.284821    4521 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f860c372-4493-4c25-b46a-a6d3af710390-workload-socket\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:03:00.285543 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:00.285391    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11b85d0f-fa82-4861-9c1c-e47a1c1c308a-istio-data" pod "11b85d0f-fa82-4861-9c1c-e47a1c1c308a" (UID: "11b85d0f-fa82-4861-9c1c-e47a1c1c308a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:03:00.285921 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:00.285890    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11b85d0f-fa82-4861-9c1c-e47a1c1c308a-workload-socket" pod "11b85d0f-fa82-4861-9c1c-e47a1c1c308a" (UID: "11b85d0f-fa82-4861-9c1c-e47a1c1c308a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:03:00.286391 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:00.286188    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d91a4f29-950e-4370-92e0-8ec493b72620-workload-socket" pod "d91a4f29-950e-4370-92e0-8ec493b72620" (UID: "d91a4f29-950e-4370-92e0-8ec493b72620"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:03:00.286391 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:00.286278    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d91a4f29-950e-4370-92e0-8ec493b72620-istio-data" pod "d91a4f29-950e-4370-92e0-8ec493b72620" (UID: "d91a4f29-950e-4370-92e0-8ec493b72620"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:03:00.286979 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:00.286595    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d91a4f29-950e-4370-92e0-8ec493b72620-istiod-ca-cert" pod "d91a4f29-950e-4370-92e0-8ec493b72620" (UID: "d91a4f29-950e-4370-92e0-8ec493b72620"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:03:00.286979 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:00.286884    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11b85d0f-fa82-4861-9c1c-e47a1c1c308a-workload-certs" pod "11b85d0f-fa82-4861-9c1c-e47a1c1c308a" (UID: "11b85d0f-fa82-4861-9c1c-e47a1c1c308a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:03:00.287568 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:00.287240    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d91a4f29-950e-4370-92e0-8ec493b72620-credential-socket" pod "d91a4f29-950e-4370-92e0-8ec493b72620" (UID: "d91a4f29-950e-4370-92e0-8ec493b72620"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:03:00.287568 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:00.287455    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11b85d0f-fa82-4861-9c1c-e47a1c1c308a-credential-socket" pod "11b85d0f-fa82-4861-9c1c-e47a1c1c308a" (UID: "11b85d0f-fa82-4861-9c1c-e47a1c1c308a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:03:00.287568 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:00.287565    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d91a4f29-950e-4370-92e0-8ec493b72620-kube-api-access-vqs62" pod "d91a4f29-950e-4370-92e0-8ec493b72620" (UID: "d91a4f29-950e-4370-92e0-8ec493b72620"). InnerVolumeSpecName "kube-api-access-vqs62". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:03:00.287856 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:00.287826    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11b85d0f-fa82-4861-9c1c-e47a1c1c308a-istiod-ca-cert" pod "11b85d0f-fa82-4861-9c1c-e47a1c1c308a" (UID: "11b85d0f-fa82-4861-9c1c-e47a1c1c308a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:03:00.288092 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:00.288068    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d91a4f29-950e-4370-92e0-8ec493b72620-workload-certs" pod "d91a4f29-950e-4370-92e0-8ec493b72620" (UID: "d91a4f29-950e-4370-92e0-8ec493b72620"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:03:00.288783 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:00.288748    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11b85d0f-fa82-4861-9c1c-e47a1c1c308a-istio-token" pod "11b85d0f-fa82-4861-9c1c-e47a1c1c308a" (UID: "11b85d0f-fa82-4861-9c1c-e47a1c1c308a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:03:00.289861 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:00.289309    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11b85d0f-fa82-4861-9c1c-e47a1c1c308a-istio-envoy" pod "11b85d0f-fa82-4861-9c1c-e47a1c1c308a" (UID: "11b85d0f-fa82-4861-9c1c-e47a1c1c308a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:03:00.290547 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:00.290509    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d91a4f29-950e-4370-92e0-8ec493b72620-istio-podinfo" pod "d91a4f29-950e-4370-92e0-8ec493b72620" (UID: "d91a4f29-950e-4370-92e0-8ec493b72620"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:03:00.290812 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:00.290787    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d91a4f29-950e-4370-92e0-8ec493b72620-istio-token" pod "d91a4f29-950e-4370-92e0-8ec493b72620" (UID: "d91a4f29-950e-4370-92e0-8ec493b72620"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:03:00.291414 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:00.291392    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/11b85d0f-fa82-4861-9c1c-e47a1c1c308a-istio-podinfo" pod "11b85d0f-fa82-4861-9c1c-e47a1c1c308a" (UID: "11b85d0f-fa82-4861-9c1c-e47a1c1c308a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:03:00.292077 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:00.292055    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11b85d0f-fa82-4861-9c1c-e47a1c1c308a-kube-api-access-m6lrz" pod "11b85d0f-fa82-4861-9c1c-e47a1c1c308a" (UID: "11b85d0f-fa82-4861-9c1c-e47a1c1c308a"). InnerVolumeSpecName "kube-api-access-m6lrz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:03:00.292500 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:00.292481    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d91a4f29-950e-4370-92e0-8ec493b72620-istio-envoy" pod "d91a4f29-950e-4370-92e0-8ec493b72620" (UID: "d91a4f29-950e-4370-92e0-8ec493b72620"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:03:00.385816 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:00.385778    4521 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6lrz\" (UniqueName: \"kubernetes.io/projected/11b85d0f-fa82-4861-9c1c-e47a1c1c308a-kube-api-access-m6lrz\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:03:00.385816 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:00.385809    4521 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11b85d0f-fa82-4861-9c1c-e47a1c1c308a-credential-socket\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:03:00.385816 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:00.385819    4521 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d91a4f29-950e-4370-92e0-8ec493b72620-istio-token\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:03:00.386114 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:00.385829    4521 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d91a4f29-950e-4370-92e0-8ec493b72620-istio-podinfo\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:03:00.386114 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:00.385840    4521 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11b85d0f-fa82-4861-9c1c-e47a1c1c308a-istio-podinfo\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:03:00.386114 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:00.385848    4521 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d91a4f29-950e-4370-92e0-8ec493b72620-istio-data\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:03:00.386114 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:00.385856    4521 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d91a4f29-950e-4370-92e0-8ec493b72620-istiod-ca-cert\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:03:00.386114 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:00.385864    4521 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11b85d0f-fa82-4861-9c1c-e47a1c1c308a-istio-data\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:03:00.386114 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:00.385872    4521 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d91a4f29-950e-4370-92e0-8ec493b72620-credential-socket\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:03:00.386114 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:00.385880    4521 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11b85d0f-fa82-4861-9c1c-e47a1c1c308a-workload-certs\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:03:00.386114 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:00.385887    4521 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d91a4f29-950e-4370-92e0-8ec493b72620-workload-certs\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:03:00.386114 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:00.385897    4521 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11b85d0f-fa82-4861-9c1c-e47a1c1c308a-istiod-ca-cert\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:03:00.386114 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:00.385906    4521 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11b85d0f-fa82-4861-9c1c-e47a1c1c308a-istio-envoy\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:03:00.386114 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:00.385915    4521 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11b85d0f-fa82-4861-9c1c-e47a1c1c308a-workload-socket\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:03:00.386114 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:00.385925    4521 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d91a4f29-950e-4370-92e0-8ec493b72620-workload-socket\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:03:00.386114 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:00.385933    4521 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11b85d0f-fa82-4861-9c1c-e47a1c1c308a-istio-token\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:03:00.386114 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:00.385943    4521 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vqs62\" (UniqueName: \"kubernetes.io/projected/d91a4f29-950e-4370-92e0-8ec493b72620-kube-api-access-vqs62\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:03:00.386114 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:00.385953    4521 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d91a4f29-950e-4370-92e0-8ec493b72620-istio-envoy\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:03:01.182789 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:01.182702    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-5tqcb"
Oct 30 21:03:01.182789 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:01.182734    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-5tqcb" event={"ID":"11b85d0f-fa82-4861-9c1c-e47a1c1c308a","Type":"ContainerDied","Data":"e44287839751a0524eb32b14aa1c652e3797b084f0a5720a9de8ff4c38451f9f"}
Oct 30 21:03:01.183261 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:01.182813    4521 scope.go:122] "RemoveContainer" containerID="b251af91f4b3645d981cdae3ca14049d5f598b6d3c79722fa36f70e80e806ab9"
Oct 30 21:03:01.184427 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:01.184409    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-jjmw4"
Oct 30 21:03:01.184503 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:01.184434    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-hlv6g"
Oct 30 21:03:01.184503 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:01.184445    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-jjmw4" event={"ID":"d91a4f29-950e-4370-92e0-8ec493b72620","Type":"ContainerDied","Data":"4061230f09c87f7aff0f0c24cd4d4ec270b1e1c6842669b401905661bb47cd7c"}
Oct 30 21:03:01.195138 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:01.195118    4521 scope.go:122] "RemoveContainer" containerID="949c736f77bbe18f449157304ec69d5f8dc30d34424b7bec3a13bef4c4ece749"
Oct 30 21:03:01.210218 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:01.210188    4521 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-hlv6g"]
Oct 30 21:03:01.215031 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:01.214998    4521 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-hlv6g"]
Oct 30 21:03:01.229272 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:01.229237    4521 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-jjmw4"]
Oct 30 21:03:01.238137 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:01.238106    4521 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-jjmw4"]
Oct 30 21:03:01.253288 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:01.253250    4521 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-5tqcb"]
Oct 30 21:03:01.264629 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:01.264599    4521 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-4a0c5b-openshift-default-676865df99-5tqcb"]
Oct 30 21:03:02.865485 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:02.858713    4521 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11b85d0f-fa82-4861-9c1c-e47a1c1c308a" path="/var/lib/kubelet/pods/11b85d0f-fa82-4861-9c1c-e47a1c1c308a/volumes"
Oct 30 21:03:02.866862 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:02.866832    4521 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d91a4f29-950e-4370-92e0-8ec493b72620" path="/var/lib/kubelet/pods/d91a4f29-950e-4370-92e0-8ec493b72620/volumes"
Oct 30 21:03:02.871916 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:02.870747    4521 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f860c372-4493-4c25-b46a-a6d3af710390" path="/var/lib/kubelet/pods/f860c372-4493-4c25-b46a-a6d3af710390/volumes"
Oct 30 21:03:07.165137 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:07.165081    4521 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-f4lmh"]
Oct 30 21:03:07.166060 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:07.166026    4521 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-f4lmh" podUID="9cb1b1d5-a595-403d-9a53-0560efac6ab7" containerName="istio-proxy" containerID="cri-o://b0bcdf545b4a406ab49a1ee5cb8638b9b0323af2615ee83b2080899fc3bddc98" gracePeriod=30
Oct 30 21:03:07.175212 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:07.175183    4521 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-6ggs9"]
Oct 30 21:03:07.175471 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:07.175447    4521 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-6ggs9" podUID="795d7e19-cf6e-4863-bf80-28ae03af6a98" containerName="istio-proxy" containerID="cri-o://8755bbaa725db4ee5c09405b8a0c9145492ebcfceba41da023778dd0787cdb8a" gracePeriod=30
Oct 30 21:03:07.185066 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:07.185034    4521 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-r586d"]
Oct 30 21:03:07.185292 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:07.185268    4521 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-r586d" podUID="69a73fe6-ec07-446c-aecc-e11806b5f492" containerName="istio-proxy" containerID="cri-o://26904e2c90ba23d519dbf5ec4696939b31e4d84322e700151eec2883e6f6a117" gracePeriod=30
Oct 30 21:03:07.573089 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:07.573046    4521 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-sbpfk"]
Oct 30 21:03:07.573371 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:07.573345    4521 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-sbpfk" podUID="7a5f91bb-7229-454f-a006-0d768f8879f4" containerName="istio-proxy" containerID="cri-o://805ef8c72c4f1574bdbfa9bbe04ce8260c0199472a8325633fd65adb6b7f588c" gracePeriod=30
Oct 30 21:03:07.601628 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:07.601588    4521 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-9zp68"]
Oct 30 21:03:07.601863 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:07.601837    4521 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-9zp68" podUID="65e4dd7b-506f-4350-a82c-e35e61706595" containerName="istio-proxy" containerID="cri-o://ed1aa36ed724c662b14ce9b03c89f27c5184ed3f8fc9e30e3fa8989d2d8ef78f" gracePeriod=30
Oct 30 21:03:07.613050 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:07.613016    4521 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-5xzm6"]
Oct 30 21:03:07.613288 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:07.613247    4521 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-5xzm6" podUID="308f74fc-a395-4ad2-8902-8ed077ab9d0d" containerName="istio-proxy" containerID="cri-o://a65276986ef16a935ea5918e247501cdea333b646757020527c9f377f598a28d" gracePeriod=30
Oct 30 21:03:07.625035 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:07.624994    4521 patch_prober.go:29] interesting pod/e2e-worker-w5-65968b-openshift-default-68896fcddc-5xzm6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 21:03:07.625179 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:07.625058    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-5xzm6" podUID="308f74fc-a395-4ad2-8902-8ed077ab9d0d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 21:03:12.373987 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:12.373682    4521 generic.go:356] "Generic (PLEG): container finished" podID="9cb1b1d5-a595-403d-9a53-0560efac6ab7" containerID="b0bcdf545b4a406ab49a1ee5cb8638b9b0323af2615ee83b2080899fc3bddc98" exitCode=0
Oct 30 21:03:12.373987 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:12.373896    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-f4lmh" event={"ID":"9cb1b1d5-a595-403d-9a53-0560efac6ab7","Type":"ContainerDied","Data":"b0bcdf545b4a406ab49a1ee5cb8638b9b0323af2615ee83b2080899fc3bddc98"}
Oct 30 21:03:12.376712 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:12.376554    4521 generic.go:356] "Generic (PLEG): container finished" podID="795d7e19-cf6e-4863-bf80-28ae03af6a98" containerID="8755bbaa725db4ee5c09405b8a0c9145492ebcfceba41da023778dd0787cdb8a" exitCode=0
Oct 30 21:03:12.376712 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:12.376604    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-6ggs9" event={"ID":"795d7e19-cf6e-4863-bf80-28ae03af6a98","Type":"ContainerDied","Data":"8755bbaa725db4ee5c09405b8a0c9145492ebcfceba41da023778dd0787cdb8a"}
Oct 30 21:03:12.378734 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:12.378575    4521 generic.go:356] "Generic (PLEG): container finished" podID="69a73fe6-ec07-446c-aecc-e11806b5f492" containerID="26904e2c90ba23d519dbf5ec4696939b31e4d84322e700151eec2883e6f6a117" exitCode=0
Oct 30 21:03:12.378734 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:12.378680    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-r586d" event={"ID":"69a73fe6-ec07-446c-aecc-e11806b5f492","Type":"ContainerDied","Data":"26904e2c90ba23d519dbf5ec4696939b31e4d84322e700151eec2883e6f6a117"}
Oct 30 21:03:12.534078 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:12.533226    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-6ggs9"
Oct 30 21:03:12.540916 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:12.540892    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-f4lmh"
Oct 30 21:03:12.543871 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:12.543848    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-r586d"
Oct 30 21:03:12.677859 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:12.677823    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69a73fe6-ec07-446c-aecc-e11806b5f492-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69a73fe6-ec07-446c-aecc-e11806b5f492-credential-socket\") pod \"69a73fe6-ec07-446c-aecc-e11806b5f492\" (UID: \"69a73fe6-ec07-446c-aecc-e11806b5f492\") "
Oct 30 21:03:12.678043 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:12.677875    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69a73fe6-ec07-446c-aecc-e11806b5f492-istio-token\" (UniqueName: \"kubernetes.io/projected/69a73fe6-ec07-446c-aecc-e11806b5f492-istio-token\") pod \"69a73fe6-ec07-446c-aecc-e11806b5f492\" (UID: \"69a73fe6-ec07-446c-aecc-e11806b5f492\") "
Oct 30 21:03:12.678043 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:12.677933    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/795d7e19-cf6e-4863-bf80-28ae03af6a98-kube-api-access-2rplp\" (UniqueName: \"kubernetes.io/projected/795d7e19-cf6e-4863-bf80-28ae03af6a98-kube-api-access-2rplp\") pod \"795d7e19-cf6e-4863-bf80-28ae03af6a98\" (UID: \"795d7e19-cf6e-4863-bf80-28ae03af6a98\") "
Oct 30 21:03:12.678043 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:12.677990    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9cb1b1d5-a595-403d-9a53-0560efac6ab7-istio-token\" (UniqueName: \"kubernetes.io/projected/9cb1b1d5-a595-403d-9a53-0560efac6ab7-istio-token\") pod \"9cb1b1d5-a595-403d-9a53-0560efac6ab7\" (UID: \"9cb1b1d5-a595-403d-9a53-0560efac6ab7\") "
Oct 30 21:03:12.678257 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:12.678034    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69a73fe6-ec07-446c-aecc-e11806b5f492-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69a73fe6-ec07-446c-aecc-e11806b5f492-istio-envoy\") pod \"69a73fe6-ec07-446c-aecc-e11806b5f492\" (UID: \"69a73fe6-ec07-446c-aecc-e11806b5f492\") "
Oct 30 21:03:12.678257 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:12.678107    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69a73fe6-ec07-446c-aecc-e11806b5f492-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69a73fe6-ec07-446c-aecc-e11806b5f492-istiod-ca-cert\") pod \"69a73fe6-ec07-446c-aecc-e11806b5f492\" (UID: \"69a73fe6-ec07-446c-aecc-e11806b5f492\") "
Oct 30 21:03:12.678257 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:12.678159    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/795d7e19-cf6e-4863-bf80-28ae03af6a98-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/795d7e19-cf6e-4863-bf80-28ae03af6a98-istio-podinfo\") pod \"795d7e19-cf6e-4863-bf80-28ae03af6a98\" (UID: \"795d7e19-cf6e-4863-bf80-28ae03af6a98\") "
Oct 30 21:03:12.678257 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:12.678239    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cb1b1d5-a595-403d-9a53-0560efac6ab7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cb1b1d5-a595-403d-9a53-0560efac6ab7-credential-socket\") pod \"9cb1b1d5-a595-403d-9a53-0560efac6ab7\" (UID: \"9cb1b1d5-a595-403d-9a53-0560efac6ab7\") "
Oct 30 21:03:12.678503 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:12.678296    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/795d7e19-cf6e-4863-bf80-28ae03af6a98-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/795d7e19-cf6e-4863-bf80-28ae03af6a98-credential-socket\") pod \"795d7e19-cf6e-4863-bf80-28ae03af6a98\" (UID: \"795d7e19-cf6e-4863-bf80-28ae03af6a98\") "
Oct 30 21:03:12.678659 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:12.678630    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69a73fe6-ec07-446c-aecc-e11806b5f492-kube-api-access-n4h24\" (UniqueName: \"kubernetes.io/projected/69a73fe6-ec07-446c-aecc-e11806b5f492-kube-api-access-n4h24\") pod \"69a73fe6-ec07-446c-aecc-e11806b5f492\" (UID: \"69a73fe6-ec07-446c-aecc-e11806b5f492\") "
Oct 30 21:03:12.678807 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:12.678779    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69a73fe6-ec07-446c-aecc-e11806b5f492-credential-socket" pod "69a73fe6-ec07-446c-aecc-e11806b5f492" (UID: "69a73fe6-ec07-446c-aecc-e11806b5f492"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:03:12.679043 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:12.679012    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69a73fe6-ec07-446c-aecc-e11806b5f492-istiod-ca-cert" pod "69a73fe6-ec07-446c-aecc-e11806b5f492" (UID: "69a73fe6-ec07-446c-aecc-e11806b5f492"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:03:12.679338 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:12.679305    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/795d7e19-cf6e-4863-bf80-28ae03af6a98-credential-socket" pod "795d7e19-cf6e-4863-bf80-28ae03af6a98" (UID: "795d7e19-cf6e-4863-bf80-28ae03af6a98"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:03:12.679432 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:12.679355    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/795d7e19-cf6e-4863-bf80-28ae03af6a98-workload-socket" pod "795d7e19-cf6e-4863-bf80-28ae03af6a98" (UID: "795d7e19-cf6e-4863-bf80-28ae03af6a98"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:03:12.679432 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:12.679394    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cb1b1d5-a595-403d-9a53-0560efac6ab7-credential-socket" pod "9cb1b1d5-a595-403d-9a53-0560efac6ab7" (UID: "9cb1b1d5-a595-403d-9a53-0560efac6ab7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:03:12.679432 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:12.678689    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/795d7e19-cf6e-4863-bf80-28ae03af6a98-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/795d7e19-cf6e-4863-bf80-28ae03af6a98-workload-socket\") pod \"795d7e19-cf6e-4863-bf80-28ae03af6a98\" (UID: \"795d7e19-cf6e-4863-bf80-28ae03af6a98\") "
Oct 30 21:03:12.679610 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:12.679466    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cb1b1d5-a595-403d-9a53-0560efac6ab7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cb1b1d5-a595-403d-9a53-0560efac6ab7-workload-socket\") pod \"9cb1b1d5-a595-403d-9a53-0560efac6ab7\" (UID: \"9cb1b1d5-a595-403d-9a53-0560efac6ab7\") "
Oct 30 21:03:12.679610 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:12.679505    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cb1b1d5-a595-403d-9a53-0560efac6ab7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9cb1b1d5-a595-403d-9a53-0560efac6ab7-istio-data\") pod \"9cb1b1d5-a595-403d-9a53-0560efac6ab7\" (UID: \"9cb1b1d5-a595-403d-9a53-0560efac6ab7\") "
Oct 30 21:03:12.679610 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:12.679566    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69a73fe6-ec07-446c-aecc-e11806b5f492-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69a73fe6-ec07-446c-aecc-e11806b5f492-workload-socket\") pod \"69a73fe6-ec07-446c-aecc-e11806b5f492\" (UID: \"69a73fe6-ec07-446c-aecc-e11806b5f492\") "
Oct 30 21:03:12.679610 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:12.679602    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/795d7e19-cf6e-4863-bf80-28ae03af6a98-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/795d7e19-cf6e-4863-bf80-28ae03af6a98-istiod-ca-cert\") pod \"795d7e19-cf6e-4863-bf80-28ae03af6a98\" (UID: \"795d7e19-cf6e-4863-bf80-28ae03af6a98\") "
Oct 30 21:03:12.679819 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:12.679634    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/795d7e19-cf6e-4863-bf80-28ae03af6a98-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/795d7e19-cf6e-4863-bf80-28ae03af6a98-workload-certs\") pod \"795d7e19-cf6e-4863-bf80-28ae03af6a98\" (UID: \"795d7e19-cf6e-4863-bf80-28ae03af6a98\") "
Oct 30 21:03:12.679819 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:12.679664    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cb1b1d5-a595-403d-9a53-0560efac6ab7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cb1b1d5-a595-403d-9a53-0560efac6ab7-workload-certs\") pod \"9cb1b1d5-a595-403d-9a53-0560efac6ab7\" (UID: \"9cb1b1d5-a595-403d-9a53-0560efac6ab7\") "
Oct 30 21:03:12.679819 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:12.679766    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/795d7e19-cf6e-4863-bf80-28ae03af6a98-istio-token\" (UniqueName: \"kubernetes.io/projected/795d7e19-cf6e-4863-bf80-28ae03af6a98-istio-token\") pod \"795d7e19-cf6e-4863-bf80-28ae03af6a98\" (UID: \"795d7e19-cf6e-4863-bf80-28ae03af6a98\") "
Oct 30 21:03:12.679819 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:12.679798    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/795d7e19-cf6e-4863-bf80-28ae03af6a98-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/795d7e19-cf6e-4863-bf80-28ae03af6a98-istio-envoy\") pod \"795d7e19-cf6e-4863-bf80-28ae03af6a98\" (UID: \"795d7e19-cf6e-4863-bf80-28ae03af6a98\") "
Oct 30 21:03:12.679819 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:12.679789    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cb1b1d5-a595-403d-9a53-0560efac6ab7-workload-socket" pod "9cb1b1d5-a595-403d-9a53-0560efac6ab7" (UID: "9cb1b1d5-a595-403d-9a53-0560efac6ab7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:03:12.680162 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:12.679835    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9cb1b1d5-a595-403d-9a53-0560efac6ab7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9cb1b1d5-a595-403d-9a53-0560efac6ab7-istio-podinfo\") pod \"9cb1b1d5-a595-403d-9a53-0560efac6ab7\" (UID: \"9cb1b1d5-a595-403d-9a53-0560efac6ab7\") "
Oct 30 21:03:12.680162 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:12.679895    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69a73fe6-ec07-446c-aecc-e11806b5f492-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69a73fe6-ec07-446c-aecc-e11806b5f492-istio-data\") pod \"69a73fe6-ec07-446c-aecc-e11806b5f492\" (UID: \"69a73fe6-ec07-446c-aecc-e11806b5f492\") "
Oct 30 21:03:12.680162 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:12.679977    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69a73fe6-ec07-446c-aecc-e11806b5f492-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69a73fe6-ec07-446c-aecc-e11806b5f492-workload-certs\") pod \"69a73fe6-ec07-446c-aecc-e11806b5f492\" (UID: \"69a73fe6-ec07-446c-aecc-e11806b5f492\") "
Oct 30 21:03:12.680162 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:12.680008    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/69a73fe6-ec07-446c-aecc-e11806b5f492-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69a73fe6-ec07-446c-aecc-e11806b5f492-istio-podinfo\") pod \"69a73fe6-ec07-446c-aecc-e11806b5f492\" (UID: \"69a73fe6-ec07-446c-aecc-e11806b5f492\") "
Oct 30 21:03:12.680162 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:12.680040    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9cb1b1d5-a595-403d-9a53-0560efac6ab7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9cb1b1d5-a595-403d-9a53-0560efac6ab7-istiod-ca-cert\") pod \"9cb1b1d5-a595-403d-9a53-0560efac6ab7\" (UID: \"9cb1b1d5-a595-403d-9a53-0560efac6ab7\") "
Oct 30 21:03:12.680162 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:12.680057    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/795d7e19-cf6e-4863-bf80-28ae03af6a98-workload-certs" pod "795d7e19-cf6e-4863-bf80-28ae03af6a98" (UID: "795d7e19-cf6e-4863-bf80-28ae03af6a98"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:03:12.680162 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:12.680091    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/795d7e19-cf6e-4863-bf80-28ae03af6a98-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/795d7e19-cf6e-4863-bf80-28ae03af6a98-istio-data\") pod \"795d7e19-cf6e-4863-bf80-28ae03af6a98\" (UID: \"795d7e19-cf6e-4863-bf80-28ae03af6a98\") "
Oct 30 21:03:12.680556 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:12.680190    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cb1b1d5-a595-403d-9a53-0560efac6ab7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9cb1b1d5-a595-403d-9a53-0560efac6ab7-istio-envoy\") pod \"9cb1b1d5-a595-403d-9a53-0560efac6ab7\" (UID: \"9cb1b1d5-a595-403d-9a53-0560efac6ab7\") "
Oct 30 21:03:12.680556 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:12.680255    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9cb1b1d5-a595-403d-9a53-0560efac6ab7-kube-api-access-6blzh\" (UniqueName: \"kubernetes.io/projected/9cb1b1d5-a595-403d-9a53-0560efac6ab7-kube-api-access-6blzh\") pod \"9cb1b1d5-a595-403d-9a53-0560efac6ab7\" (UID: \"9cb1b1d5-a595-403d-9a53-0560efac6ab7\") "
Oct 30 21:03:12.680556 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:12.680278    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69a73fe6-ec07-446c-aecc-e11806b5f492-workload-socket" pod "69a73fe6-ec07-446c-aecc-e11806b5f492" (UID: "69a73fe6-ec07-446c-aecc-e11806b5f492"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:03:12.680734 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:12.680705    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/795d7e19-cf6e-4863-bf80-28ae03af6a98-istiod-ca-cert" pod "795d7e19-cf6e-4863-bf80-28ae03af6a98" (UID: "795d7e19-cf6e-4863-bf80-28ae03af6a98"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:03:12.681272 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:12.681248    4521 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69a73fe6-ec07-446c-aecc-e11806b5f492-credential-socket\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:03:12.681375 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:12.681280    4521 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69a73fe6-ec07-446c-aecc-e11806b5f492-istiod-ca-cert\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:03:12.681375 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:12.681311    4521 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cb1b1d5-a595-403d-9a53-0560efac6ab7-credential-socket\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:03:12.681375 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:12.681340    4521 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/795d7e19-cf6e-4863-bf80-28ae03af6a98-credential-socket\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:03:12.681375 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:12.681354    4521 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/795d7e19-cf6e-4863-bf80-28ae03af6a98-workload-socket\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:03:12.681607 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:12.681370    4521 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cb1b1d5-a595-403d-9a53-0560efac6ab7-workload-socket\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:03:12.681607 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:12.681397    4521 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69a73fe6-ec07-446c-aecc-e11806b5f492-workload-socket\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:03:12.681607 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:12.681411    4521 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/795d7e19-cf6e-4863-bf80-28ae03af6a98-istiod-ca-cert\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:03:12.681607 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:12.681425    4521 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/795d7e19-cf6e-4863-bf80-28ae03af6a98-workload-certs\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:03:12.681607 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:12.681485    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cb1b1d5-a595-403d-9a53-0560efac6ab7-istio-data" pod "9cb1b1d5-a595-403d-9a53-0560efac6ab7" (UID: "9cb1b1d5-a595-403d-9a53-0560efac6ab7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:03:12.681607 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:12.681598    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/795d7e19-cf6e-4863-bf80-28ae03af6a98-kube-api-access-2rplp" pod "795d7e19-cf6e-4863-bf80-28ae03af6a98" (UID: "795d7e19-cf6e-4863-bf80-28ae03af6a98"). InnerVolumeSpecName "kube-api-access-2rplp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:03:12.683151 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:12.683120    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69a73fe6-ec07-446c-aecc-e11806b5f492-workload-certs" pod "69a73fe6-ec07-446c-aecc-e11806b5f492" (UID: "69a73fe6-ec07-446c-aecc-e11806b5f492"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:03:12.684625 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:12.684596    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69a73fe6-ec07-446c-aecc-e11806b5f492-istio-envoy" pod "69a73fe6-ec07-446c-aecc-e11806b5f492" (UID: "69a73fe6-ec07-446c-aecc-e11806b5f492"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:03:12.684715 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:12.684682    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69a73fe6-ec07-446c-aecc-e11806b5f492-kube-api-access-n4h24" pod "69a73fe6-ec07-446c-aecc-e11806b5f492" (UID: "69a73fe6-ec07-446c-aecc-e11806b5f492"). InnerVolumeSpecName "kube-api-access-n4h24". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:03:12.684894 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:12.684816    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cb1b1d5-a595-403d-9a53-0560efac6ab7-istio-token" pod "9cb1b1d5-a595-403d-9a53-0560efac6ab7" (UID: "9cb1b1d5-a595-403d-9a53-0560efac6ab7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:03:12.684998 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:12.684962    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cb1b1d5-a595-403d-9a53-0560efac6ab7-workload-certs" pod "9cb1b1d5-a595-403d-9a53-0560efac6ab7" (UID: "9cb1b1d5-a595-403d-9a53-0560efac6ab7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:03:12.685069 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:12.684994    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/795d7e19-cf6e-4863-bf80-28ae03af6a98-istio-envoy" pod "795d7e19-cf6e-4863-bf80-28ae03af6a98" (UID: "795d7e19-cf6e-4863-bf80-28ae03af6a98"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:03:12.685069 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:12.685043    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/795d7e19-cf6e-4863-bf80-28ae03af6a98-istio-data" pod "795d7e19-cf6e-4863-bf80-28ae03af6a98" (UID: "795d7e19-cf6e-4863-bf80-28ae03af6a98"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:03:12.685296 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:12.685178    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69a73fe6-ec07-446c-aecc-e11806b5f492-istio-data" pod "69a73fe6-ec07-446c-aecc-e11806b5f492" (UID: "69a73fe6-ec07-446c-aecc-e11806b5f492"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:03:12.685468 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:12.685341    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cb1b1d5-a595-403d-9a53-0560efac6ab7-istiod-ca-cert" pod "9cb1b1d5-a595-403d-9a53-0560efac6ab7" (UID: "9cb1b1d5-a595-403d-9a53-0560efac6ab7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:03:12.685926 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:12.685803    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cb1b1d5-a595-403d-9a53-0560efac6ab7-kube-api-access-6blzh" pod "9cb1b1d5-a595-403d-9a53-0560efac6ab7" (UID: "9cb1b1d5-a595-403d-9a53-0560efac6ab7"). InnerVolumeSpecName "kube-api-access-6blzh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:03:12.686114 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:12.686003    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/69a73fe6-ec07-446c-aecc-e11806b5f492-istio-podinfo" pod "69a73fe6-ec07-446c-aecc-e11806b5f492" (UID: "69a73fe6-ec07-446c-aecc-e11806b5f492"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:03:12.686572 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:12.686552    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/795d7e19-cf6e-4863-bf80-28ae03af6a98-istio-podinfo" pod "795d7e19-cf6e-4863-bf80-28ae03af6a98" (UID: "795d7e19-cf6e-4863-bf80-28ae03af6a98"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:03:12.687043 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:12.687011    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69a73fe6-ec07-446c-aecc-e11806b5f492-istio-token" pod "69a73fe6-ec07-446c-aecc-e11806b5f492" (UID: "69a73fe6-ec07-446c-aecc-e11806b5f492"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:03:12.687401 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:12.687376    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/795d7e19-cf6e-4863-bf80-28ae03af6a98-istio-token" pod "795d7e19-cf6e-4863-bf80-28ae03af6a98" (UID: "795d7e19-cf6e-4863-bf80-28ae03af6a98"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:03:12.688431 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:12.688399    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9cb1b1d5-a595-403d-9a53-0560efac6ab7-istio-podinfo" pod "9cb1b1d5-a595-403d-9a53-0560efac6ab7" (UID: "9cb1b1d5-a595-403d-9a53-0560efac6ab7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:03:12.689007 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:12.688985    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cb1b1d5-a595-403d-9a53-0560efac6ab7-istio-envoy" pod "9cb1b1d5-a595-403d-9a53-0560efac6ab7" (UID: "9cb1b1d5-a595-403d-9a53-0560efac6ab7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:03:12.783086 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:12.783050    4521 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cb1b1d5-a595-403d-9a53-0560efac6ab7-workload-certs\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:03:12.783086 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:12.783093    4521 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/795d7e19-cf6e-4863-bf80-28ae03af6a98-istio-token\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:03:12.783297 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:12.783110    4521 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/795d7e19-cf6e-4863-bf80-28ae03af6a98-istio-envoy\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:03:12.783297 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:12.783123    4521 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9cb1b1d5-a595-403d-9a53-0560efac6ab7-istio-podinfo\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:03:12.783297 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:12.783140    4521 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69a73fe6-ec07-446c-aecc-e11806b5f492-istio-data\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:03:12.783297 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:12.783153    4521 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69a73fe6-ec07-446c-aecc-e11806b5f492-workload-certs\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:03:12.783297 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:12.783167    4521 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69a73fe6-ec07-446c-aecc-e11806b5f492-istio-podinfo\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:03:12.783297 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:12.783179    4521 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9cb1b1d5-a595-403d-9a53-0560efac6ab7-istiod-ca-cert\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:03:12.783297 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:12.783192    4521 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/795d7e19-cf6e-4863-bf80-28ae03af6a98-istio-data\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:03:12.783297 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:12.783206    4521 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9cb1b1d5-a595-403d-9a53-0560efac6ab7-istio-envoy\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:03:12.783297 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:12.783220    4521 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6blzh\" (UniqueName: \"kubernetes.io/projected/9cb1b1d5-a595-403d-9a53-0560efac6ab7-kube-api-access-6blzh\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:03:12.783297 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:12.783235    4521 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69a73fe6-ec07-446c-aecc-e11806b5f492-istio-token\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:03:12.783297 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:12.783247    4521 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rplp\" (UniqueName: \"kubernetes.io/projected/795d7e19-cf6e-4863-bf80-28ae03af6a98-kube-api-access-2rplp\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:03:12.783297 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:12.783262    4521 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9cb1b1d5-a595-403d-9a53-0560efac6ab7-istio-token\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:03:12.783297 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:12.783274    4521 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69a73fe6-ec07-446c-aecc-e11806b5f492-istio-envoy\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:03:12.783297 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:12.783287    4521 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/795d7e19-cf6e-4863-bf80-28ae03af6a98-istio-podinfo\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:03:12.783297 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:12.783300    4521 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4h24\" (UniqueName: \"kubernetes.io/projected/69a73fe6-ec07-446c-aecc-e11806b5f492-kube-api-access-n4h24\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:03:12.784126 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:12.783313    4521 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9cb1b1d5-a595-403d-9a53-0560efac6ab7-istio-data\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:03:12.903729 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:12.903685    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-9zp68"
Oct 30 21:03:12.962894 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:12.962815    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-5xzm6"
Oct 30 21:03:12.966674 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:12.966646    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-sbpfk"
Oct 30 21:03:12.985159 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:12.985134    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65e4dd7b-506f-4350-a82c-e35e61706595-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65e4dd7b-506f-4350-a82c-e35e61706595-workload-certs\") pod \"65e4dd7b-506f-4350-a82c-e35e61706595\" (UID: \"65e4dd7b-506f-4350-a82c-e35e61706595\") "
Oct 30 21:03:12.985315 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:12.985194    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65e4dd7b-506f-4350-a82c-e35e61706595-kube-api-access-mm9tf\" (UniqueName: \"kubernetes.io/projected/65e4dd7b-506f-4350-a82c-e35e61706595-kube-api-access-mm9tf\") pod \"65e4dd7b-506f-4350-a82c-e35e61706595\" (UID: \"65e4dd7b-506f-4350-a82c-e35e61706595\") "
Oct 30 21:03:12.985315 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:12.985222    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65e4dd7b-506f-4350-a82c-e35e61706595-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65e4dd7b-506f-4350-a82c-e35e61706595-credential-socket\") pod \"65e4dd7b-506f-4350-a82c-e35e61706595\" (UID: \"65e4dd7b-506f-4350-a82c-e35e61706595\") "
Oct 30 21:03:12.985315 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:12.985251    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/65e4dd7b-506f-4350-a82c-e35e61706595-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65e4dd7b-506f-4350-a82c-e35e61706595-istio-podinfo\") pod \"65e4dd7b-506f-4350-a82c-e35e61706595\" (UID: \"65e4dd7b-506f-4350-a82c-e35e61706595\") "
Oct 30 21:03:12.985489 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:12.985349    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65e4dd7b-506f-4350-a82c-e35e61706595-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65e4dd7b-506f-4350-a82c-e35e61706595-workload-socket\") pod \"65e4dd7b-506f-4350-a82c-e35e61706595\" (UID: \"65e4dd7b-506f-4350-a82c-e35e61706595\") "
Oct 30 21:03:12.985489 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:12.985428    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65e4dd7b-506f-4350-a82c-e35e61706595-istio-token\" (UniqueName: \"kubernetes.io/projected/65e4dd7b-506f-4350-a82c-e35e61706595-istio-token\") pod \"65e4dd7b-506f-4350-a82c-e35e61706595\" (UID: \"65e4dd7b-506f-4350-a82c-e35e61706595\") "
Oct 30 21:03:12.985489 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:12.985462    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65e4dd7b-506f-4350-a82c-e35e61706595-workload-certs" pod "65e4dd7b-506f-4350-a82c-e35e61706595" (UID: "65e4dd7b-506f-4350-a82c-e35e61706595"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:03:12.985683 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:12.985510    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65e4dd7b-506f-4350-a82c-e35e61706595-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65e4dd7b-506f-4350-a82c-e35e61706595-istio-data\") pod \"65e4dd7b-506f-4350-a82c-e35e61706595\" (UID: \"65e4dd7b-506f-4350-a82c-e35e61706595\") "
Oct 30 21:03:12.985683 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:12.985554    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65e4dd7b-506f-4350-a82c-e35e61706595-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65e4dd7b-506f-4350-a82c-e35e61706595-istiod-ca-cert\") pod \"65e4dd7b-506f-4350-a82c-e35e61706595\" (UID: \"65e4dd7b-506f-4350-a82c-e35e61706595\") "
Oct 30 21:03:12.985683 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:12.985580    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65e4dd7b-506f-4350-a82c-e35e61706595-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65e4dd7b-506f-4350-a82c-e35e61706595-istio-envoy\") pod \"65e4dd7b-506f-4350-a82c-e35e61706595\" (UID: \"65e4dd7b-506f-4350-a82c-e35e61706595\") "
Oct 30 21:03:12.985683 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:12.985592    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65e4dd7b-506f-4350-a82c-e35e61706595-credential-socket" pod "65e4dd7b-506f-4350-a82c-e35e61706595" (UID: "65e4dd7b-506f-4350-a82c-e35e61706595"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:03:12.985683 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:12.985614    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65e4dd7b-506f-4350-a82c-e35e61706595-workload-socket" pod "65e4dd7b-506f-4350-a82c-e35e61706595" (UID: "65e4dd7b-506f-4350-a82c-e35e61706595"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:03:12.985936 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:12.985804    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65e4dd7b-506f-4350-a82c-e35e61706595-istio-data" pod "65e4dd7b-506f-4350-a82c-e35e61706595" (UID: "65e4dd7b-506f-4350-a82c-e35e61706595"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:03:12.985936 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:12.985895    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65e4dd7b-506f-4350-a82c-e35e61706595-istiod-ca-cert" pod "65e4dd7b-506f-4350-a82c-e35e61706595" (UID: "65e4dd7b-506f-4350-a82c-e35e61706595"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:03:12.986095 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:12.986033    4521 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65e4dd7b-506f-4350-a82c-e35e61706595-workload-socket\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:03:12.986095 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:12.986050    4521 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65e4dd7b-506f-4350-a82c-e35e61706595-istio-data\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:03:12.986095 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:12.986059    4521 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65e4dd7b-506f-4350-a82c-e35e61706595-istiod-ca-cert\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:03:12.986095 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:12.986067    4521 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65e4dd7b-506f-4350-a82c-e35e61706595-workload-certs\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:03:12.986095 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:12.986077    4521 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65e4dd7b-506f-4350-a82c-e35e61706595-credential-socket\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:03:12.988131 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:12.988107    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/65e4dd7b-506f-4350-a82c-e35e61706595-istio-podinfo" pod "65e4dd7b-506f-4350-a82c-e35e61706595" (UID: "65e4dd7b-506f-4350-a82c-e35e61706595"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:03:12.988229 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:12.988161    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65e4dd7b-506f-4350-a82c-e35e61706595-istio-token" pod "65e4dd7b-506f-4350-a82c-e35e61706595" (UID: "65e4dd7b-506f-4350-a82c-e35e61706595"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:03:12.988229 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:12.988196    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65e4dd7b-506f-4350-a82c-e35e61706595-kube-api-access-mm9tf" pod "65e4dd7b-506f-4350-a82c-e35e61706595" (UID: "65e4dd7b-506f-4350-a82c-e35e61706595"). InnerVolumeSpecName "kube-api-access-mm9tf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:03:12.988342 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:12.988303    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65e4dd7b-506f-4350-a82c-e35e61706595-istio-envoy" pod "65e4dd7b-506f-4350-a82c-e35e61706595" (UID: "65e4dd7b-506f-4350-a82c-e35e61706595"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:03:13.086780 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:13.086692    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/308f74fc-a395-4ad2-8902-8ed077ab9d0d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/308f74fc-a395-4ad2-8902-8ed077ab9d0d-istio-envoy\") pod \"308f74fc-a395-4ad2-8902-8ed077ab9d0d\" (UID: \"308f74fc-a395-4ad2-8902-8ed077ab9d0d\") "
Oct 30 21:03:13.086780 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:13.086735    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/308f74fc-a395-4ad2-8902-8ed077ab9d0d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/308f74fc-a395-4ad2-8902-8ed077ab9d0d-workload-socket\") pod \"308f74fc-a395-4ad2-8902-8ed077ab9d0d\" (UID: \"308f74fc-a395-4ad2-8902-8ed077ab9d0d\") "
Oct 30 21:03:13.087018 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:13.086794    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a5f91bb-7229-454f-a006-0d768f8879f4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a5f91bb-7229-454f-a006-0d768f8879f4-istio-data\") pod \"7a5f91bb-7229-454f-a006-0d768f8879f4\" (UID: \"7a5f91bb-7229-454f-a006-0d768f8879f4\") "
Oct 30 21:03:13.087018 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:13.086822    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/308f74fc-a395-4ad2-8902-8ed077ab9d0d-istio-token\" (UniqueName: \"kubernetes.io/projected/308f74fc-a395-4ad2-8902-8ed077ab9d0d-istio-token\") pod \"308f74fc-a395-4ad2-8902-8ed077ab9d0d\" (UID: \"308f74fc-a395-4ad2-8902-8ed077ab9d0d\") "
Oct 30 21:03:13.087018 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:13.086848    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a5f91bb-7229-454f-a006-0d768f8879f4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a5f91bb-7229-454f-a006-0d768f8879f4-istio-envoy\") pod \"7a5f91bb-7229-454f-a006-0d768f8879f4\" (UID: \"7a5f91bb-7229-454f-a006-0d768f8879f4\") "
Oct 30 21:03:13.087018 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:13.086884    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7a5f91bb-7229-454f-a006-0d768f8879f4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a5f91bb-7229-454f-a006-0d768f8879f4-istio-podinfo\") pod \"7a5f91bb-7229-454f-a006-0d768f8879f4\" (UID: \"7a5f91bb-7229-454f-a006-0d768f8879f4\") "
Oct 30 21:03:13.087018 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:13.086914    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/308f74fc-a395-4ad2-8902-8ed077ab9d0d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/308f74fc-a395-4ad2-8902-8ed077ab9d0d-credential-socket\") pod \"308f74fc-a395-4ad2-8902-8ed077ab9d0d\" (UID: \"308f74fc-a395-4ad2-8902-8ed077ab9d0d\") "
Oct 30 21:03:13.087018 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:13.086959    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/308f74fc-a395-4ad2-8902-8ed077ab9d0d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/308f74fc-a395-4ad2-8902-8ed077ab9d0d-istio-data\") pod \"308f74fc-a395-4ad2-8902-8ed077ab9d0d\" (UID: \"308f74fc-a395-4ad2-8902-8ed077ab9d0d\") "
Oct 30 21:03:13.087018 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:13.086989    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a5f91bb-7229-454f-a006-0d768f8879f4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a5f91bb-7229-454f-a006-0d768f8879f4-workload-certs\") pod \"7a5f91bb-7229-454f-a006-0d768f8879f4\" (UID: \"7a5f91bb-7229-454f-a006-0d768f8879f4\") "
Oct 30 21:03:13.087018 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:13.087006    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a5f91bb-7229-454f-a006-0d768f8879f4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a5f91bb-7229-454f-a006-0d768f8879f4-workload-socket\") pod \"7a5f91bb-7229-454f-a006-0d768f8879f4\" (UID: \"7a5f91bb-7229-454f-a006-0d768f8879f4\") "
Oct 30 21:03:13.087465 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:13.087044    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/308f74fc-a395-4ad2-8902-8ed077ab9d0d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/308f74fc-a395-4ad2-8902-8ed077ab9d0d-istiod-ca-cert\") pod \"308f74fc-a395-4ad2-8902-8ed077ab9d0d\" (UID: \"308f74fc-a395-4ad2-8902-8ed077ab9d0d\") "
Oct 30 21:03:13.087465 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:13.087063    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a5f91bb-7229-454f-a006-0d768f8879f4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a5f91bb-7229-454f-a006-0d768f8879f4-credential-socket\") pod \"7a5f91bb-7229-454f-a006-0d768f8879f4\" (UID: \"7a5f91bb-7229-454f-a006-0d768f8879f4\") "
Oct 30 21:03:13.087465 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:13.087085    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/308f74fc-a395-4ad2-8902-8ed077ab9d0d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/308f74fc-a395-4ad2-8902-8ed077ab9d0d-workload-certs\") pod \"308f74fc-a395-4ad2-8902-8ed077ab9d0d\" (UID: \"308f74fc-a395-4ad2-8902-8ed077ab9d0d\") "
Oct 30 21:03:13.087465 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:13.087109    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a5f91bb-7229-454f-a006-0d768f8879f4-istio-token\" (UniqueName: \"kubernetes.io/projected/7a5f91bb-7229-454f-a006-0d768f8879f4-istio-token\") pod \"7a5f91bb-7229-454f-a006-0d768f8879f4\" (UID: \"7a5f91bb-7229-454f-a006-0d768f8879f4\") "
Oct 30 21:03:13.087465 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:13.087162    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/308f74fc-a395-4ad2-8902-8ed077ab9d0d-kube-api-access-p6mh4\" (UniqueName: \"kubernetes.io/projected/308f74fc-a395-4ad2-8902-8ed077ab9d0d-kube-api-access-p6mh4\") pod \"308f74fc-a395-4ad2-8902-8ed077ab9d0d\" (UID: \"308f74fc-a395-4ad2-8902-8ed077ab9d0d\") "
Oct 30 21:03:13.087465 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:13.087194    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a5f91bb-7229-454f-a006-0d768f8879f4-kube-api-access-w84kk\" (UniqueName: \"kubernetes.io/projected/7a5f91bb-7229-454f-a006-0d768f8879f4-kube-api-access-w84kk\") pod \"7a5f91bb-7229-454f-a006-0d768f8879f4\" (UID: \"7a5f91bb-7229-454f-a006-0d768f8879f4\") "
Oct 30 21:03:13.087465 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:13.087218    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/308f74fc-a395-4ad2-8902-8ed077ab9d0d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/308f74fc-a395-4ad2-8902-8ed077ab9d0d-istio-podinfo\") pod \"308f74fc-a395-4ad2-8902-8ed077ab9d0d\" (UID: \"308f74fc-a395-4ad2-8902-8ed077ab9d0d\") "
Oct 30 21:03:13.087465 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:13.087241    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a5f91bb-7229-454f-a006-0d768f8879f4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a5f91bb-7229-454f-a006-0d768f8879f4-istiod-ca-cert\") pod \"7a5f91bb-7229-454f-a006-0d768f8879f4\" (UID: \"7a5f91bb-7229-454f-a006-0d768f8879f4\") "
Oct 30 21:03:13.087892 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:13.087643    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/308f74fc-a395-4ad2-8902-8ed077ab9d0d-workload-certs" pod "308f74fc-a395-4ad2-8902-8ed077ab9d0d" (UID: "308f74fc-a395-4ad2-8902-8ed077ab9d0d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:03:13.087892 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:13.087650    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a5f91bb-7229-454f-a006-0d768f8879f4-workload-certs" pod "7a5f91bb-7229-454f-a006-0d768f8879f4" (UID: "7a5f91bb-7229-454f-a006-0d768f8879f4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:03:13.087892 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:13.087668    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/308f74fc-a395-4ad2-8902-8ed077ab9d0d-workload-socket" pod "308f74fc-a395-4ad2-8902-8ed077ab9d0d" (UID: "308f74fc-a395-4ad2-8902-8ed077ab9d0d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:03:13.087892 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:13.087879    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/308f74fc-a395-4ad2-8902-8ed077ab9d0d-credential-socket" pod "308f74fc-a395-4ad2-8902-8ed077ab9d0d" (UID: "308f74fc-a395-4ad2-8902-8ed077ab9d0d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:03:13.088169 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:13.088015    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a5f91bb-7229-454f-a006-0d768f8879f4-workload-socket" pod "7a5f91bb-7229-454f-a006-0d768f8879f4" (UID: "7a5f91bb-7229-454f-a006-0d768f8879f4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:03:13.088169 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:13.088051    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/308f74fc-a395-4ad2-8902-8ed077ab9d0d-istio-data" pod "308f74fc-a395-4ad2-8902-8ed077ab9d0d" (UID: "308f74fc-a395-4ad2-8902-8ed077ab9d0d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:03:13.088273 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:13.088212    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a5f91bb-7229-454f-a006-0d768f8879f4-credential-socket" pod "7a5f91bb-7229-454f-a006-0d768f8879f4" (UID: "7a5f91bb-7229-454f-a006-0d768f8879f4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:03:13.088489 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:13.088468    4521 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/308f74fc-a395-4ad2-8902-8ed077ab9d0d-credential-socket\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:03:13.088573 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:13.088494    4521 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/308f74fc-a395-4ad2-8902-8ed077ab9d0d-istio-data\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:03:13.088573 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:13.088511    4521 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a5f91bb-7229-454f-a006-0d768f8879f4-workload-certs\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:03:13.088573 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:13.088515    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a5f91bb-7229-454f-a006-0d768f8879f4-istiod-ca-cert" pod "7a5f91bb-7229-454f-a006-0d768f8879f4" (UID: "7a5f91bb-7229-454f-a006-0d768f8879f4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:03:13.088573 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:13.088524    4521 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a5f91bb-7229-454f-a006-0d768f8879f4-workload-socket\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:03:13.088573 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:13.088541    4521 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mm9tf\" (UniqueName: \"kubernetes.io/projected/65e4dd7b-506f-4350-a82c-e35e61706595-kube-api-access-mm9tf\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:03:13.088573 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:13.088556    4521 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a5f91bb-7229-454f-a006-0d768f8879f4-credential-socket\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:03:13.088573 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:13.088570    4521 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65e4dd7b-506f-4350-a82c-e35e61706595-istio-podinfo\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:03:13.088933 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:13.088585    4521 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/308f74fc-a395-4ad2-8902-8ed077ab9d0d-workload-certs\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:03:13.088933 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:13.088601    4521 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65e4dd7b-506f-4350-a82c-e35e61706595-istio-token\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:03:13.088933 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:13.088614    4521 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/308f74fc-a395-4ad2-8902-8ed077ab9d0d-workload-socket\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:03:13.088933 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:13.088627    4521 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65e4dd7b-506f-4350-a82c-e35e61706595-istio-envoy\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:03:13.088933 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:13.088723    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a5f91bb-7229-454f-a006-0d768f8879f4-istio-data" pod "7a5f91bb-7229-454f-a006-0d768f8879f4" (UID: "7a5f91bb-7229-454f-a006-0d768f8879f4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:03:13.089161 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:13.088973    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/308f74fc-a395-4ad2-8902-8ed077ab9d0d-istiod-ca-cert" pod "308f74fc-a395-4ad2-8902-8ed077ab9d0d" (UID: "308f74fc-a395-4ad2-8902-8ed077ab9d0d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:03:13.091072 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:13.091031    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/308f74fc-a395-4ad2-8902-8ed077ab9d0d-kube-api-access-p6mh4" pod "308f74fc-a395-4ad2-8902-8ed077ab9d0d" (UID: "308f74fc-a395-4ad2-8902-8ed077ab9d0d"). InnerVolumeSpecName "kube-api-access-p6mh4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:03:13.091501 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:13.091434    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7a5f91bb-7229-454f-a006-0d768f8879f4-istio-podinfo" pod "7a5f91bb-7229-454f-a006-0d768f8879f4" (UID: "7a5f91bb-7229-454f-a006-0d768f8879f4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:03:13.091762 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:13.091688    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/308f74fc-a395-4ad2-8902-8ed077ab9d0d-istio-envoy" pod "308f74fc-a395-4ad2-8902-8ed077ab9d0d" (UID: "308f74fc-a395-4ad2-8902-8ed077ab9d0d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:03:13.091762 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:13.091725    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a5f91bb-7229-454f-a006-0d768f8879f4-kube-api-access-w84kk" pod "7a5f91bb-7229-454f-a006-0d768f8879f4" (UID: "7a5f91bb-7229-454f-a006-0d768f8879f4"). InnerVolumeSpecName "kube-api-access-w84kk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:03:13.091918 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:13.091765    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/308f74fc-a395-4ad2-8902-8ed077ab9d0d-istio-token" pod "308f74fc-a395-4ad2-8902-8ed077ab9d0d" (UID: "308f74fc-a395-4ad2-8902-8ed077ab9d0d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:03:13.092413 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:13.092386    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/308f74fc-a395-4ad2-8902-8ed077ab9d0d-istio-podinfo" pod "308f74fc-a395-4ad2-8902-8ed077ab9d0d" (UID: "308f74fc-a395-4ad2-8902-8ed077ab9d0d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:03:13.092618 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:13.092598    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a5f91bb-7229-454f-a006-0d768f8879f4-istio-token" pod "7a5f91bb-7229-454f-a006-0d768f8879f4" (UID: "7a5f91bb-7229-454f-a006-0d768f8879f4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:03:13.093498 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:13.093480    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a5f91bb-7229-454f-a006-0d768f8879f4-istio-envoy" pod "7a5f91bb-7229-454f-a006-0d768f8879f4" (UID: "7a5f91bb-7229-454f-a006-0d768f8879f4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:03:13.189860 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:13.189816    4521 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/308f74fc-a395-4ad2-8902-8ed077ab9d0d-istiod-ca-cert\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:03:13.189860 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:13.189845    4521 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a5f91bb-7229-454f-a006-0d768f8879f4-istio-token\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:03:13.189860 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:13.189856    4521 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6mh4\" (UniqueName: \"kubernetes.io/projected/308f74fc-a395-4ad2-8902-8ed077ab9d0d-kube-api-access-p6mh4\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:03:13.189860 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:13.189866    4521 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w84kk\" (UniqueName: \"kubernetes.io/projected/7a5f91bb-7229-454f-a006-0d768f8879f4-kube-api-access-w84kk\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:03:13.190224 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:13.189875    4521 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/308f74fc-a395-4ad2-8902-8ed077ab9d0d-istio-podinfo\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:03:13.190224 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:13.189884    4521 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a5f91bb-7229-454f-a006-0d768f8879f4-istiod-ca-cert\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:03:13.190224 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:13.189894    4521 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/308f74fc-a395-4ad2-8902-8ed077ab9d0d-istio-envoy\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:03:13.190224 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:13.189902    4521 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a5f91bb-7229-454f-a006-0d768f8879f4-istio-data\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:03:13.190224 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:13.189910    4521 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/308f74fc-a395-4ad2-8902-8ed077ab9d0d-istio-token\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:03:13.190224 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:13.189918    4521 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a5f91bb-7229-454f-a006-0d768f8879f4-istio-envoy\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:03:13.190224 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:13.189925    4521 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a5f91bb-7229-454f-a006-0d768f8879f4-istio-podinfo\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:03:13.387689 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:13.387608    4521 generic.go:356] "Generic (PLEG): container finished" podID="65e4dd7b-506f-4350-a82c-e35e61706595" containerID="ed1aa36ed724c662b14ce9b03c89f27c5184ed3f8fc9e30e3fa8989d2d8ef78f" exitCode=0
Oct 30 21:03:13.387689 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:13.387661    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-9zp68"
Oct 30 21:03:13.388146 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:13.387691    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-9zp68" event={"ID":"65e4dd7b-506f-4350-a82c-e35e61706595","Type":"ContainerDied","Data":"ed1aa36ed724c662b14ce9b03c89f27c5184ed3f8fc9e30e3fa8989d2d8ef78f"}
Oct 30 21:03:13.388146 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:13.387748    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-9zp68" event={"ID":"65e4dd7b-506f-4350-a82c-e35e61706595","Type":"ContainerDied","Data":"95716966944a5650b4f979d900f41ff438b9d14efdad462e0f8ccb968509e906"}
Oct 30 21:03:13.388146 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:13.387774    4521 scope.go:122] "RemoveContainer" containerID="ed1aa36ed724c662b14ce9b03c89f27c5184ed3f8fc9e30e3fa8989d2d8ef78f"
Oct 30 21:03:13.389296 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:13.389276    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-r586d"
Oct 30 21:03:13.389425 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:13.389346    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-r586d" event={"ID":"69a73fe6-ec07-446c-aecc-e11806b5f492","Type":"ContainerDied","Data":"db80057a710ee0a6d92e9cff3022f79cfc41fd58188303461a09d3983b8ce3fa"}
Oct 30 21:03:13.391850 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:13.391803    4521 generic.go:356] "Generic (PLEG): container finished" podID="7a5f91bb-7229-454f-a006-0d768f8879f4" containerID="805ef8c72c4f1574bdbfa9bbe04ce8260c0199472a8325633fd65adb6b7f588c" exitCode=0
Oct 30 21:03:13.391998 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:13.391864    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-sbpfk"
Oct 30 21:03:13.391998 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:13.391889    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-sbpfk" event={"ID":"7a5f91bb-7229-454f-a006-0d768f8879f4","Type":"ContainerDied","Data":"805ef8c72c4f1574bdbfa9bbe04ce8260c0199472a8325633fd65adb6b7f588c"}
Oct 30 21:03:13.391998 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:13.391935    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-sbpfk" event={"ID":"7a5f91bb-7229-454f-a006-0d768f8879f4","Type":"ContainerDied","Data":"03fcb95c00ed7b9c5b71c9219fac7330d82cfd09d91d63a03fce277ebf015d09"}
Oct 30 21:03:13.393638 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:13.393618    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-f4lmh"
Oct 30 21:03:13.393739 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:13.393660    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-f4lmh" event={"ID":"9cb1b1d5-a595-403d-9a53-0560efac6ab7","Type":"ContainerDied","Data":"2692070e31abaf16b6b68ccdfbbd671354a0d45df2119badad4bc0ba3ae0fab5"}
Oct 30 21:03:13.395616 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:13.395281    4521 generic.go:356] "Generic (PLEG): container finished" podID="308f74fc-a395-4ad2-8902-8ed077ab9d0d" containerID="a65276986ef16a935ea5918e247501cdea333b646757020527c9f377f598a28d" exitCode=0
Oct 30 21:03:13.395616 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:13.395341    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-5xzm6"
Oct 30 21:03:13.395616 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:13.395346    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-5xzm6" event={"ID":"308f74fc-a395-4ad2-8902-8ed077ab9d0d","Type":"ContainerDied","Data":"a65276986ef16a935ea5918e247501cdea333b646757020527c9f377f598a28d"}
Oct 30 21:03:13.395616 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:13.395394    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-5xzm6" event={"ID":"308f74fc-a395-4ad2-8902-8ed077ab9d0d","Type":"ContainerDied","Data":"41eb1e7c07f2a5aaee86262aed569817507742dc9db250aa90e7946a2278f6f0"}
Oct 30 21:03:13.399369 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:13.398312    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-6ggs9" event={"ID":"795d7e19-cf6e-4863-bf80-28ae03af6a98","Type":"ContainerDied","Data":"36f90eb210934c1392c67db8ba77f05d87b19e7f7edc61f966e956794558ee49"}
Oct 30 21:03:13.399369 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:13.398415    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-6ggs9"
Oct 30 21:03:13.402043 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:13.401968    4521 scope.go:122] "RemoveContainer" containerID="ed1aa36ed724c662b14ce9b03c89f27c5184ed3f8fc9e30e3fa8989d2d8ef78f"
Oct 30 21:03:13.402586 ip-10-0-131-125 kubenswrapper[4521]: E1030 21:03:13.402562    4521 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed1aa36ed724c662b14ce9b03c89f27c5184ed3f8fc9e30e3fa8989d2d8ef78f\": container with ID starting with ed1aa36ed724c662b14ce9b03c89f27c5184ed3f8fc9e30e3fa8989d2d8ef78f not found: ID does not exist" containerID="ed1aa36ed724c662b14ce9b03c89f27c5184ed3f8fc9e30e3fa8989d2d8ef78f"
Oct 30 21:03:13.402759 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:13.402597    4521 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed1aa36ed724c662b14ce9b03c89f27c5184ed3f8fc9e30e3fa8989d2d8ef78f"} err="failed to get container status \"ed1aa36ed724c662b14ce9b03c89f27c5184ed3f8fc9e30e3fa8989d2d8ef78f\": rpc error: code = NotFound desc = could not find container \"ed1aa36ed724c662b14ce9b03c89f27c5184ed3f8fc9e30e3fa8989d2d8ef78f\": container with ID starting with ed1aa36ed724c662b14ce9b03c89f27c5184ed3f8fc9e30e3fa8989d2d8ef78f not found: ID does not exist"
Oct 30 21:03:13.402759 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:13.402619    4521 scope.go:122] "RemoveContainer" containerID="26904e2c90ba23d519dbf5ec4696939b31e4d84322e700151eec2883e6f6a117"
Oct 30 21:03:13.416842 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:13.416818    4521 scope.go:122] "RemoveContainer" containerID="805ef8c72c4f1574bdbfa9bbe04ce8260c0199472a8325633fd65adb6b7f588c"
Oct 30 21:03:13.422102 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:13.422061    4521 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-r586d"]
Oct 30 21:03:13.430694 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:13.430666    4521 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-r586d"]
Oct 30 21:03:13.433359 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:13.433234    4521 scope.go:122] "RemoveContainer" containerID="805ef8c72c4f1574bdbfa9bbe04ce8260c0199472a8325633fd65adb6b7f588c"
Oct 30 21:03:13.433982 ip-10-0-131-125 kubenswrapper[4521]: E1030 21:03:13.433630    4521 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"805ef8c72c4f1574bdbfa9bbe04ce8260c0199472a8325633fd65adb6b7f588c\": container with ID starting with 805ef8c72c4f1574bdbfa9bbe04ce8260c0199472a8325633fd65adb6b7f588c not found: ID does not exist" containerID="805ef8c72c4f1574bdbfa9bbe04ce8260c0199472a8325633fd65adb6b7f588c"
Oct 30 21:03:13.433982 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:13.433670    4521 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"805ef8c72c4f1574bdbfa9bbe04ce8260c0199472a8325633fd65adb6b7f588c"} err="failed to get container status \"805ef8c72c4f1574bdbfa9bbe04ce8260c0199472a8325633fd65adb6b7f588c\": rpc error: code = NotFound desc = could not find container \"805ef8c72c4f1574bdbfa9bbe04ce8260c0199472a8325633fd65adb6b7f588c\": container with ID starting with 805ef8c72c4f1574bdbfa9bbe04ce8260c0199472a8325633fd65adb6b7f588c not found: ID does not exist"
Oct 30 21:03:13.433982 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:13.433697    4521 scope.go:122] "RemoveContainer" containerID="b0bcdf545b4a406ab49a1ee5cb8638b9b0323af2615ee83b2080899fc3bddc98"
Oct 30 21:03:13.449120 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:13.449083    4521 scope.go:122] "RemoveContainer" containerID="a65276986ef16a935ea5918e247501cdea333b646757020527c9f377f598a28d"
Oct 30 21:03:13.458700 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:13.458631    4521 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-6ggs9"]
Oct 30 21:03:13.463012 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:13.462988    4521 scope.go:122] "RemoveContainer" containerID="a65276986ef16a935ea5918e247501cdea333b646757020527c9f377f598a28d"
Oct 30 21:03:13.463357 ip-10-0-131-125 kubenswrapper[4521]: E1030 21:03:13.463306    4521 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a65276986ef16a935ea5918e247501cdea333b646757020527c9f377f598a28d\": container with ID starting with a65276986ef16a935ea5918e247501cdea333b646757020527c9f377f598a28d not found: ID does not exist" containerID="a65276986ef16a935ea5918e247501cdea333b646757020527c9f377f598a28d"
Oct 30 21:03:13.463357 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:13.463349    4521 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-6ggs9"]
Oct 30 21:03:13.463470 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:13.463366    4521 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a65276986ef16a935ea5918e247501cdea333b646757020527c9f377f598a28d"} err="failed to get container status \"a65276986ef16a935ea5918e247501cdea333b646757020527c9f377f598a28d\": rpc error: code = NotFound desc = could not find container \"a65276986ef16a935ea5918e247501cdea333b646757020527c9f377f598a28d\": container with ID starting with a65276986ef16a935ea5918e247501cdea333b646757020527c9f377f598a28d not found: ID does not exist"
Oct 30 21:03:13.463470 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:13.463385    4521 scope.go:122] "RemoveContainer" containerID="8755bbaa725db4ee5c09405b8a0c9145492ebcfceba41da023778dd0787cdb8a"
Oct 30 21:03:13.480891 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:13.480861    4521 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-5xzm6"]
Oct 30 21:03:13.485050 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:13.485019    4521 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-5xzm6"]
Oct 30 21:03:13.504000 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:13.503961    4521 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-f4lmh"]
Oct 30 21:03:13.513441 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:13.513411    4521 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-c2d1eb-openshift-default-ddff45b59-f4lmh"]
Oct 30 21:03:13.540226 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:13.540188    4521 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-sbpfk"]
Oct 30 21:03:13.552999 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:13.552967    4521 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-sbpfk"]
Oct 30 21:03:13.567680 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:13.567646    4521 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-9zp68"]
Oct 30 21:03:13.581594 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:13.581555    4521 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-65968b-openshift-default-68896fcddc-9zp68"]
Oct 30 21:03:14.792447 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:14.792414    4521 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="308f74fc-a395-4ad2-8902-8ed077ab9d0d" path="/var/lib/kubelet/pods/308f74fc-a395-4ad2-8902-8ed077ab9d0d/volumes"
Oct 30 21:03:14.792897 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:14.792884    4521 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65e4dd7b-506f-4350-a82c-e35e61706595" path="/var/lib/kubelet/pods/65e4dd7b-506f-4350-a82c-e35e61706595/volumes"
Oct 30 21:03:14.793290 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:14.793278    4521 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69a73fe6-ec07-446c-aecc-e11806b5f492" path="/var/lib/kubelet/pods/69a73fe6-ec07-446c-aecc-e11806b5f492/volumes"
Oct 30 21:03:14.794025 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:14.794011    4521 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="795d7e19-cf6e-4863-bf80-28ae03af6a98" path="/var/lib/kubelet/pods/795d7e19-cf6e-4863-bf80-28ae03af6a98/volumes"
Oct 30 21:03:14.794432 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:14.794420    4521 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a5f91bb-7229-454f-a006-0d768f8879f4" path="/var/lib/kubelet/pods/7a5f91bb-7229-454f-a006-0d768f8879f4/volumes"
Oct 30 21:03:14.794815 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:14.794803    4521 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9cb1b1d5-a595-403d-9a53-0560efac6ab7" path="/var/lib/kubelet/pods/9cb1b1d5-a595-403d-9a53-0560efac6ab7/volumes"
Oct 30 21:03:15.274043 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:15.274008    4521 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-m9wn4"]
Oct 30 21:03:15.274264 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:15.274240    4521 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-m9wn4" podUID="9dc1356e-e12d-49f9-822e-f1ccf4fdaa3b" containerName="istio-proxy" containerID="cri-o://e420f18aec5b42561ae1b8c5b8056f63c7fbabe169afbf6204220c200e21f9aa" gracePeriod=30
Oct 30 21:03:15.304330 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:15.304281    4521 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-w5lzs"]
Oct 30 21:03:15.304577 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:15.304551    4521 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-w5lzs" podUID="b7f18233-f19c-4f74-a071-62d63402691c" containerName="istio-proxy" containerID="cri-o://cf3a9a5ec8ba245d0d64dedab1e441bbfab2abe751df9baeae4e64280ecf8c4f" gracePeriod=30
Oct 30 21:03:15.308264 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:15.308238    4521 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-nvwt8"]
Oct 30 21:03:15.308454 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:15.308434    4521 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-nvwt8" podUID="6cff2b61-dcbd-4450-b776-43eff57a4fd2" containerName="istio-proxy" containerID="cri-o://d0cd9f072652f1ce279084eed6d6e0420305cd803eafc75e89d9ff4abb5a09c5" gracePeriod=30
Oct 30 21:03:16.432662 ip-10-0-131-125 kubenswrapper[4521]: E1030 21:03:16.432621    4521 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lkn2k" containerName="extractor"
Oct 30 21:03:20.297765 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:20.297733    4521 patch_prober.go:29] interesting pod/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-nvwt8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 21:03:20.298149 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:20.297789    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-nvwt8" podUID="6cff2b61-dcbd-4450-b776-43eff57a4fd2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 21:03:20.437357 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:20.437235    4521 patch_prober.go:29] interesting pod/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-m9wn4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Oct 30 21:03:20.437357 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:20.437317    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-m9wn4" podUID="9dc1356e-e12d-49f9-822e-f1ccf4fdaa3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Oct 30 21:03:20.474148 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:20.474119    4521 generic.go:356] "Generic (PLEG): container finished" podID="6cff2b61-dcbd-4450-b776-43eff57a4fd2" containerID="d0cd9f072652f1ce279084eed6d6e0420305cd803eafc75e89d9ff4abb5a09c5" exitCode=0
Oct 30 21:03:20.474312 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:20.474186    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-nvwt8" event={"ID":"6cff2b61-dcbd-4450-b776-43eff57a4fd2","Type":"ContainerDied","Data":"d0cd9f072652f1ce279084eed6d6e0420305cd803eafc75e89d9ff4abb5a09c5"}
Oct 30 21:03:20.476694 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:20.476076    4521 generic.go:356] "Generic (PLEG): container finished" podID="9dc1356e-e12d-49f9-822e-f1ccf4fdaa3b" containerID="e420f18aec5b42561ae1b8c5b8056f63c7fbabe169afbf6204220c200e21f9aa" exitCode=0
Oct 30 21:03:20.476694 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:20.476150    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-m9wn4" event={"ID":"9dc1356e-e12d-49f9-822e-f1ccf4fdaa3b","Type":"ContainerDied","Data":"e420f18aec5b42561ae1b8c5b8056f63c7fbabe169afbf6204220c200e21f9aa"}
Oct 30 21:03:20.478976 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:20.478951    4521 generic.go:356] "Generic (PLEG): container finished" podID="b7f18233-f19c-4f74-a071-62d63402691c" containerID="cf3a9a5ec8ba245d0d64dedab1e441bbfab2abe751df9baeae4e64280ecf8c4f" exitCode=0
Oct 30 21:03:20.479123 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:20.479097    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-w5lzs" event={"ID":"b7f18233-f19c-4f74-a071-62d63402691c","Type":"ContainerDied","Data":"cf3a9a5ec8ba245d0d64dedab1e441bbfab2abe751df9baeae4e64280ecf8c4f"}
Oct 30 21:03:20.586091 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:20.586058    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-w5lzs"
Oct 30 21:03:20.629976 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:20.629905    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-m9wn4"
Oct 30 21:03:20.644745 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:20.644723    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-nvwt8"
Oct 30 21:03:20.771623 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:20.771585    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7f18233-f19c-4f74-a071-62d63402691c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7f18233-f19c-4f74-a071-62d63402691c-istiod-ca-cert\") pod \"b7f18233-f19c-4f74-a071-62d63402691c\" (UID: \"b7f18233-f19c-4f74-a071-62d63402691c\") "
Oct 30 21:03:20.771845 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:20.771635    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9dc1356e-e12d-49f9-822e-f1ccf4fdaa3b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dc1356e-e12d-49f9-822e-f1ccf4fdaa3b-credential-socket\") pod \"9dc1356e-e12d-49f9-822e-f1ccf4fdaa3b\" (UID: \"9dc1356e-e12d-49f9-822e-f1ccf4fdaa3b\") "
Oct 30 21:03:20.771845 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:20.771666    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6cff2b61-dcbd-4450-b776-43eff57a4fd2-kube-api-access-bgwgj\" (UniqueName: \"kubernetes.io/projected/6cff2b61-dcbd-4450-b776-43eff57a4fd2-kube-api-access-bgwgj\") pod \"6cff2b61-dcbd-4450-b776-43eff57a4fd2\" (UID: \"6cff2b61-dcbd-4450-b776-43eff57a4fd2\") "
Oct 30 21:03:20.771845 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:20.771703    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7f18233-f19c-4f74-a071-62d63402691c-kube-api-access-kbkf5\" (UniqueName: \"kubernetes.io/projected/b7f18233-f19c-4f74-a071-62d63402691c-kube-api-access-kbkf5\") pod \"b7f18233-f19c-4f74-a071-62d63402691c\" (UID: \"b7f18233-f19c-4f74-a071-62d63402691c\") "
Oct 30 21:03:20.771845 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:20.771745    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6cff2b61-dcbd-4450-b776-43eff57a4fd2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cff2b61-dcbd-4450-b776-43eff57a4fd2-workload-socket\") pod \"6cff2b61-dcbd-4450-b776-43eff57a4fd2\" (UID: \"6cff2b61-dcbd-4450-b776-43eff57a4fd2\") "
Oct 30 21:03:20.771845 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:20.771790    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6cff2b61-dcbd-4450-b776-43eff57a4fd2-istio-token\" (UniqueName: \"kubernetes.io/projected/6cff2b61-dcbd-4450-b776-43eff57a4fd2-istio-token\") pod \"6cff2b61-dcbd-4450-b776-43eff57a4fd2\" (UID: \"6cff2b61-dcbd-4450-b776-43eff57a4fd2\") "
Oct 30 21:03:20.771845 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:20.771827    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6cff2b61-dcbd-4450-b776-43eff57a4fd2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6cff2b61-dcbd-4450-b776-43eff57a4fd2-workload-certs\") pod \"6cff2b61-dcbd-4450-b776-43eff57a4fd2\" (UID: \"6cff2b61-dcbd-4450-b776-43eff57a4fd2\") "
Oct 30 21:03:20.772229 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:20.771857    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9dc1356e-e12d-49f9-822e-f1ccf4fdaa3b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9dc1356e-e12d-49f9-822e-f1ccf4fdaa3b-istiod-ca-cert\") pod \"9dc1356e-e12d-49f9-822e-f1ccf4fdaa3b\" (UID: \"9dc1356e-e12d-49f9-822e-f1ccf4fdaa3b\") "
Oct 30 21:03:20.772229 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:20.771888    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9dc1356e-e12d-49f9-822e-f1ccf4fdaa3b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9dc1356e-e12d-49f9-822e-f1ccf4fdaa3b-istio-envoy\") pod \"9dc1356e-e12d-49f9-822e-f1ccf4fdaa3b\" (UID: \"9dc1356e-e12d-49f9-822e-f1ccf4fdaa3b\") "
Oct 30 21:03:20.772229 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:20.771916    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6cff2b61-dcbd-4450-b776-43eff57a4fd2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6cff2b61-dcbd-4450-b776-43eff57a4fd2-istio-data\") pod \"6cff2b61-dcbd-4450-b776-43eff57a4fd2\" (UID: \"6cff2b61-dcbd-4450-b776-43eff57a4fd2\") "
Oct 30 21:03:20.772229 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:20.771954    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dc1356e-e12d-49f9-822e-f1ccf4fdaa3b-credential-socket" pod "9dc1356e-e12d-49f9-822e-f1ccf4fdaa3b" (UID: "9dc1356e-e12d-49f9-822e-f1ccf4fdaa3b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:03:20.772229 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:20.771966    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6cff2b61-dcbd-4450-b776-43eff57a4fd2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6cff2b61-dcbd-4450-b776-43eff57a4fd2-istio-podinfo\") pod \"6cff2b61-dcbd-4450-b776-43eff57a4fd2\" (UID: \"6cff2b61-dcbd-4450-b776-43eff57a4fd2\") "
Oct 30 21:03:20.772229 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:20.771991    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7f18233-f19c-4f74-a071-62d63402691c-istiod-ca-cert" pod "b7f18233-f19c-4f74-a071-62d63402691c" (UID: "b7f18233-f19c-4f74-a071-62d63402691c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:03:20.772229 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:20.772028    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7f18233-f19c-4f74-a071-62d63402691c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7f18233-f19c-4f74-a071-62d63402691c-workload-socket\") pod \"b7f18233-f19c-4f74-a071-62d63402691c\" (UID: \"b7f18233-f19c-4f74-a071-62d63402691c\") "
Oct 30 21:03:20.772229 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:20.772080    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6cff2b61-dcbd-4450-b776-43eff57a4fd2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6cff2b61-dcbd-4450-b776-43eff57a4fd2-istio-envoy\") pod \"6cff2b61-dcbd-4450-b776-43eff57a4fd2\" (UID: \"6cff2b61-dcbd-4450-b776-43eff57a4fd2\") "
Oct 30 21:03:20.772229 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:20.772135    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7f18233-f19c-4f74-a071-62d63402691c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7f18233-f19c-4f74-a071-62d63402691c-istio-envoy\") pod \"b7f18233-f19c-4f74-a071-62d63402691c\" (UID: \"b7f18233-f19c-4f74-a071-62d63402691c\") "
Oct 30 21:03:20.772229 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:20.772160    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7f18233-f19c-4f74-a071-62d63402691c-istio-token\" (UniqueName: \"kubernetes.io/projected/b7f18233-f19c-4f74-a071-62d63402691c-istio-token\") pod \"b7f18233-f19c-4f74-a071-62d63402691c\" (UID: \"b7f18233-f19c-4f74-a071-62d63402691c\") "
Oct 30 21:03:20.772229 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:20.772186    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cff2b61-dcbd-4450-b776-43eff57a4fd2-workload-certs" pod "6cff2b61-dcbd-4450-b776-43eff57a4fd2" (UID: "6cff2b61-dcbd-4450-b776-43eff57a4fd2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:03:20.772229 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:20.772205    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6cff2b61-dcbd-4450-b776-43eff57a4fd2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cff2b61-dcbd-4450-b776-43eff57a4fd2-credential-socket\") pod \"6cff2b61-dcbd-4450-b776-43eff57a4fd2\" (UID: \"6cff2b61-dcbd-4450-b776-43eff57a4fd2\") "
Oct 30 21:03:20.772980 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:20.772248    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6cff2b61-dcbd-4450-b776-43eff57a4fd2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6cff2b61-dcbd-4450-b776-43eff57a4fd2-istiod-ca-cert\") pod \"6cff2b61-dcbd-4450-b776-43eff57a4fd2\" (UID: \"6cff2b61-dcbd-4450-b776-43eff57a4fd2\") "
Oct 30 21:03:20.772980 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:20.772280    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9dc1356e-e12d-49f9-822e-f1ccf4fdaa3b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dc1356e-e12d-49f9-822e-f1ccf4fdaa3b-workload-socket\") pod \"9dc1356e-e12d-49f9-822e-f1ccf4fdaa3b\" (UID: \"9dc1356e-e12d-49f9-822e-f1ccf4fdaa3b\") "
Oct 30 21:03:20.772980 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:20.772307    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9dc1356e-e12d-49f9-822e-f1ccf4fdaa3b-istio-token\" (UniqueName: \"kubernetes.io/projected/9dc1356e-e12d-49f9-822e-f1ccf4fdaa3b-istio-token\") pod \"9dc1356e-e12d-49f9-822e-f1ccf4fdaa3b\" (UID: \"9dc1356e-e12d-49f9-822e-f1ccf4fdaa3b\") "
Oct 30 21:03:20.772980 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:20.772372    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9dc1356e-e12d-49f9-822e-f1ccf4fdaa3b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9dc1356e-e12d-49f9-822e-f1ccf4fdaa3b-istio-data\") pod \"9dc1356e-e12d-49f9-822e-f1ccf4fdaa3b\" (UID: \"9dc1356e-e12d-49f9-822e-f1ccf4fdaa3b\") "
Oct 30 21:03:20.772980 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:20.772407    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7f18233-f19c-4f74-a071-62d63402691c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7f18233-f19c-4f74-a071-62d63402691c-credential-socket\") pod \"b7f18233-f19c-4f74-a071-62d63402691c\" (UID: \"b7f18233-f19c-4f74-a071-62d63402691c\") "
Oct 30 21:03:20.772980 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:20.772425    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cff2b61-dcbd-4450-b776-43eff57a4fd2-workload-socket" pod "6cff2b61-dcbd-4450-b776-43eff57a4fd2" (UID: "6cff2b61-dcbd-4450-b776-43eff57a4fd2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:03:20.772980 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:20.772440    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7f18233-f19c-4f74-a071-62d63402691c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7f18233-f19c-4f74-a071-62d63402691c-workload-certs\") pod \"b7f18233-f19c-4f74-a071-62d63402691c\" (UID: \"b7f18233-f19c-4f74-a071-62d63402691c\") "
Oct 30 21:03:20.772980 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:20.772467    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7f18233-f19c-4f74-a071-62d63402691c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7f18233-f19c-4f74-a071-62d63402691c-istio-data\") pod \"b7f18233-f19c-4f74-a071-62d63402691c\" (UID: \"b7f18233-f19c-4f74-a071-62d63402691c\") "
Oct 30 21:03:20.772980 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:20.772499    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9dc1356e-e12d-49f9-822e-f1ccf4fdaa3b-kube-api-access-4cc4j\" (UniqueName: \"kubernetes.io/projected/9dc1356e-e12d-49f9-822e-f1ccf4fdaa3b-kube-api-access-4cc4j\") pod \"9dc1356e-e12d-49f9-822e-f1ccf4fdaa3b\" (UID: \"9dc1356e-e12d-49f9-822e-f1ccf4fdaa3b\") "
Oct 30 21:03:20.772980 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:20.772538    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9dc1356e-e12d-49f9-822e-f1ccf4fdaa3b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9dc1356e-e12d-49f9-822e-f1ccf4fdaa3b-workload-certs\") pod \"9dc1356e-e12d-49f9-822e-f1ccf4fdaa3b\" (UID: \"9dc1356e-e12d-49f9-822e-f1ccf4fdaa3b\") "
Oct 30 21:03:20.772980 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:20.772563    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cff2b61-dcbd-4450-b776-43eff57a4fd2-istio-data" pod "6cff2b61-dcbd-4450-b776-43eff57a4fd2" (UID: "6cff2b61-dcbd-4450-b776-43eff57a4fd2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:03:20.772980 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:20.772581    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b7f18233-f19c-4f74-a071-62d63402691c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7f18233-f19c-4f74-a071-62d63402691c-istio-podinfo\") pod \"b7f18233-f19c-4f74-a071-62d63402691c\" (UID: \"b7f18233-f19c-4f74-a071-62d63402691c\") "
Oct 30 21:03:20.772980 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:20.772612    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9dc1356e-e12d-49f9-822e-f1ccf4fdaa3b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9dc1356e-e12d-49f9-822e-f1ccf4fdaa3b-istio-podinfo\") pod \"9dc1356e-e12d-49f9-822e-f1ccf4fdaa3b\" (UID: \"9dc1356e-e12d-49f9-822e-f1ccf4fdaa3b\") "
Oct 30 21:03:20.772980 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:20.772639    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9dc1356e-e12d-49f9-822e-f1ccf4fdaa3b-istiod-ca-cert" pod "9dc1356e-e12d-49f9-822e-f1ccf4fdaa3b" (UID: "9dc1356e-e12d-49f9-822e-f1ccf4fdaa3b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:03:20.772980 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:20.772770    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dc1356e-e12d-49f9-822e-f1ccf4fdaa3b-workload-socket" pod "9dc1356e-e12d-49f9-822e-f1ccf4fdaa3b" (UID: "9dc1356e-e12d-49f9-822e-f1ccf4fdaa3b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:03:20.772980 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:20.772899    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7f18233-f19c-4f74-a071-62d63402691c-workload-certs" pod "b7f18233-f19c-4f74-a071-62d63402691c" (UID: "b7f18233-f19c-4f74-a071-62d63402691c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:03:20.773870 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:20.773060    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7f18233-f19c-4f74-a071-62d63402691c-workload-socket" pod "b7f18233-f19c-4f74-a071-62d63402691c" (UID: "b7f18233-f19c-4f74-a071-62d63402691c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:03:20.773945 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:20.773924    4521 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7f18233-f19c-4f74-a071-62d63402691c-istiod-ca-cert\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:03:20.774006 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:20.773952    4521 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dc1356e-e12d-49f9-822e-f1ccf4fdaa3b-credential-socket\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:03:20.774006 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:20.773968    4521 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cff2b61-dcbd-4450-b776-43eff57a4fd2-workload-socket\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:03:20.774006 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:20.773984    4521 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6cff2b61-dcbd-4450-b776-43eff57a4fd2-workload-certs\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:03:20.774006 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:20.773997    4521 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9dc1356e-e12d-49f9-822e-f1ccf4fdaa3b-istiod-ca-cert\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:03:20.774223 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:20.774014    4521 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6cff2b61-dcbd-4450-b776-43eff57a4fd2-istio-data\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:03:20.774223 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:20.774028    4521 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7f18233-f19c-4f74-a071-62d63402691c-workload-socket\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:03:20.774223 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:20.774043    4521 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dc1356e-e12d-49f9-822e-f1ccf4fdaa3b-workload-socket\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:03:20.774223 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:20.774072    4521 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7f18233-f19c-4f74-a071-62d63402691c-workload-certs\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:03:20.775058 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:20.774663    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dc1356e-e12d-49f9-822e-f1ccf4fdaa3b-istio-envoy" pod "9dc1356e-e12d-49f9-822e-f1ccf4fdaa3b" (UID: "9dc1356e-e12d-49f9-822e-f1ccf4fdaa3b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:03:20.775058 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:20.774717    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cff2b61-dcbd-4450-b776-43eff57a4fd2-kube-api-access-bgwgj" pod "6cff2b61-dcbd-4450-b776-43eff57a4fd2" (UID: "6cff2b61-dcbd-4450-b776-43eff57a4fd2"). InnerVolumeSpecName "kube-api-access-bgwgj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:03:20.775058 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:20.774951    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dc1356e-e12d-49f9-822e-f1ccf4fdaa3b-workload-certs" pod "9dc1356e-e12d-49f9-822e-f1ccf4fdaa3b" (UID: "9dc1356e-e12d-49f9-822e-f1ccf4fdaa3b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:03:20.775564 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:20.775538    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9dc1356e-e12d-49f9-822e-f1ccf4fdaa3b-istio-podinfo" pod "9dc1356e-e12d-49f9-822e-f1ccf4fdaa3b" (UID: "9dc1356e-e12d-49f9-822e-f1ccf4fdaa3b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:03:20.775893 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:20.775868    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cff2b61-dcbd-4450-b776-43eff57a4fd2-istio-token" pod "6cff2b61-dcbd-4450-b776-43eff57a4fd2" (UID: "6cff2b61-dcbd-4450-b776-43eff57a4fd2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:03:20.776230 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:20.776204    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7f18233-f19c-4f74-a071-62d63402691c-kube-api-access-kbkf5" pod "b7f18233-f19c-4f74-a071-62d63402691c" (UID: "b7f18233-f19c-4f74-a071-62d63402691c"). InnerVolumeSpecName "kube-api-access-kbkf5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:03:20.776572 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:20.776547    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9dc1356e-e12d-49f9-822e-f1ccf4fdaa3b-istio-token" pod "9dc1356e-e12d-49f9-822e-f1ccf4fdaa3b" (UID: "9dc1356e-e12d-49f9-822e-f1ccf4fdaa3b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:03:20.776864 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:20.776842    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7f18233-f19c-4f74-a071-62d63402691c-credential-socket" pod "b7f18233-f19c-4f74-a071-62d63402691c" (UID: "b7f18233-f19c-4f74-a071-62d63402691c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:03:20.777056 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:20.777034    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7f18233-f19c-4f74-a071-62d63402691c-istio-data" pod "b7f18233-f19c-4f74-a071-62d63402691c" (UID: "b7f18233-f19c-4f74-a071-62d63402691c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:03:20.777196 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:20.777168    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dc1356e-e12d-49f9-822e-f1ccf4fdaa3b-istio-data" pod "9dc1356e-e12d-49f9-822e-f1ccf4fdaa3b" (UID: "9dc1356e-e12d-49f9-822e-f1ccf4fdaa3b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:03:20.777270 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:20.777248    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b7f18233-f19c-4f74-a071-62d63402691c-istio-podinfo" pod "b7f18233-f19c-4f74-a071-62d63402691c" (UID: "b7f18233-f19c-4f74-a071-62d63402691c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:03:20.777387 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:20.777260    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cff2b61-dcbd-4450-b776-43eff57a4fd2-credential-socket" pod "6cff2b61-dcbd-4450-b776-43eff57a4fd2" (UID: "6cff2b61-dcbd-4450-b776-43eff57a4fd2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:03:20.777718 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:20.777689    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6cff2b61-dcbd-4450-b776-43eff57a4fd2-istio-podinfo" pod "6cff2b61-dcbd-4450-b776-43eff57a4fd2" (UID: "6cff2b61-dcbd-4450-b776-43eff57a4fd2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:03:20.777718 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:20.777733    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6cff2b61-dcbd-4450-b776-43eff57a4fd2-istiod-ca-cert" pod "6cff2b61-dcbd-4450-b776-43eff57a4fd2" (UID: "6cff2b61-dcbd-4450-b776-43eff57a4fd2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:03:20.778262 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:20.777964    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cff2b61-dcbd-4450-b776-43eff57a4fd2-istio-envoy" pod "6cff2b61-dcbd-4450-b776-43eff57a4fd2" (UID: "6cff2b61-dcbd-4450-b776-43eff57a4fd2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:03:20.778262 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:20.778249    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7f18233-f19c-4f74-a071-62d63402691c-istio-envoy" pod "b7f18233-f19c-4f74-a071-62d63402691c" (UID: "b7f18233-f19c-4f74-a071-62d63402691c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:03:20.778414 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:20.778391    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9dc1356e-e12d-49f9-822e-f1ccf4fdaa3b-kube-api-access-4cc4j" pod "9dc1356e-e12d-49f9-822e-f1ccf4fdaa3b" (UID: "9dc1356e-e12d-49f9-822e-f1ccf4fdaa3b"). InnerVolumeSpecName "kube-api-access-4cc4j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:03:20.778490 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:20.778472    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7f18233-f19c-4f74-a071-62d63402691c-istio-token" pod "b7f18233-f19c-4f74-a071-62d63402691c" (UID: "b7f18233-f19c-4f74-a071-62d63402691c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:03:20.874710 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:20.874670    4521 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6cff2b61-dcbd-4450-b776-43eff57a4fd2-istio-envoy\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:03:20.874710 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:20.874704    4521 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7f18233-f19c-4f74-a071-62d63402691c-istio-envoy\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:03:20.874710 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:20.874714    4521 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7f18233-f19c-4f74-a071-62d63402691c-istio-token\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:03:20.874936 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:20.874724    4521 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cff2b61-dcbd-4450-b776-43eff57a4fd2-credential-socket\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:03:20.874936 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:20.874733    4521 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6cff2b61-dcbd-4450-b776-43eff57a4fd2-istiod-ca-cert\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:03:20.874936 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:20.874742    4521 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9dc1356e-e12d-49f9-822e-f1ccf4fdaa3b-istio-token\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:03:20.874936 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:20.874750    4521 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9dc1356e-e12d-49f9-822e-f1ccf4fdaa3b-istio-data\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:03:20.874936 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:20.874757    4521 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7f18233-f19c-4f74-a071-62d63402691c-credential-socket\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:03:20.874936 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:20.874765    4521 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7f18233-f19c-4f74-a071-62d63402691c-istio-data\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:03:20.874936 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:20.874774    4521 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4cc4j\" (UniqueName: \"kubernetes.io/projected/9dc1356e-e12d-49f9-822e-f1ccf4fdaa3b-kube-api-access-4cc4j\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:03:20.874936 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:20.874784    4521 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9dc1356e-e12d-49f9-822e-f1ccf4fdaa3b-workload-certs\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:03:20.874936 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:20.874791    4521 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7f18233-f19c-4f74-a071-62d63402691c-istio-podinfo\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:03:20.874936 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:20.874799    4521 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9dc1356e-e12d-49f9-822e-f1ccf4fdaa3b-istio-podinfo\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:03:20.874936 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:20.874807    4521 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bgwgj\" (UniqueName: \"kubernetes.io/projected/6cff2b61-dcbd-4450-b776-43eff57a4fd2-kube-api-access-bgwgj\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:03:20.874936 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:20.874815    4521 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbkf5\" (UniqueName: \"kubernetes.io/projected/b7f18233-f19c-4f74-a071-62d63402691c-kube-api-access-kbkf5\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:03:20.874936 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:20.874823    4521 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6cff2b61-dcbd-4450-b776-43eff57a4fd2-istio-token\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:03:20.874936 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:20.874831    4521 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9dc1356e-e12d-49f9-822e-f1ccf4fdaa3b-istio-envoy\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:03:20.874936 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:20.874838    4521 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6cff2b61-dcbd-4450-b776-43eff57a4fd2-istio-podinfo\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:03:21.488624 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:21.488580    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-m9wn4" event={"ID":"9dc1356e-e12d-49f9-822e-f1ccf4fdaa3b","Type":"ContainerDied","Data":"9530665c69204811e9a52e6b2ac4dbc591bb52a655d56f2dfe0be2f7bf3b5cf1"}
Oct 30 21:03:21.489119 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:21.488597    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-m9wn4"
Oct 30 21:03:21.489119 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:21.488685    4521 scope.go:122] "RemoveContainer" containerID="e420f18aec5b42561ae1b8c5b8056f63c7fbabe169afbf6204220c200e21f9aa"
Oct 30 21:03:21.490237 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:21.490220    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-w5lzs"
Oct 30 21:03:21.490316 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:21.490215    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-w5lzs" event={"ID":"b7f18233-f19c-4f74-a071-62d63402691c","Type":"ContainerDied","Data":"781e27a63bceb65a8f8fe6fd87bd3a8f35913ae7e2e9e996a42424b6c18300d9"}
Oct 30 21:03:21.492012 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:21.491986    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-nvwt8" event={"ID":"6cff2b61-dcbd-4450-b776-43eff57a4fd2","Type":"ContainerDied","Data":"8985bf0af536f344755ad49bc04b93d5a08db8a67e0b9ce4571c5f0c4080ec4a"}
Oct 30 21:03:21.492117 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:21.492028    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-nvwt8"
Oct 30 21:03:21.501037 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:21.501012    4521 scope.go:122] "RemoveContainer" containerID="cf3a9a5ec8ba245d0d64dedab1e441bbfab2abe751df9baeae4e64280ecf8c4f"
Oct 30 21:03:21.514007 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:21.513981    4521 scope.go:122] "RemoveContainer" containerID="d0cd9f072652f1ce279084eed6d6e0420305cd803eafc75e89d9ff4abb5a09c5"
Oct 30 21:03:21.532771 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:21.532741    4521 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-m9wn4"]
Oct 30 21:03:21.544874 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:21.544844    4521 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-m9wn4"]
Oct 30 21:03:21.568229 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:21.568158    4521 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-nvwt8"]
Oct 30 21:03:21.593027 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:21.592988    4521 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-nvwt8"]
Oct 30 21:03:21.620595 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:21.620555    4521 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-w5lzs"]
Oct 30 21:03:21.626693 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:21.626664    4521 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-5a8f94-openshift-default-75588f8b8f-w5lzs"]
Oct 30 21:03:22.791122 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:22.791078    4521 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6cff2b61-dcbd-4450-b776-43eff57a4fd2" path="/var/lib/kubelet/pods/6cff2b61-dcbd-4450-b776-43eff57a4fd2/volumes"
Oct 30 21:03:22.791753 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:22.791732    4521 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9dc1356e-e12d-49f9-822e-f1ccf4fdaa3b" path="/var/lib/kubelet/pods/9dc1356e-e12d-49f9-822e-f1ccf4fdaa3b/volumes"
Oct 30 21:03:22.792187 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:22.792168    4521 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7f18233-f19c-4f74-a071-62d63402691c" path="/var/lib/kubelet/pods/b7f18233-f19c-4f74-a071-62d63402691c/volumes"
Oct 30 21:03:22.875564 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:22.875522    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4kxb"]
Oct 30 21:03:29.400302 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:29.400254    4521 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-c4b379/distinct-2-w1-c4b379-kserve-7999b785f5-97fz5"]
Oct 30 21:03:29.419108 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:29.419077    4521 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-c4b379/premium-w1-c4b379-kserve-575f896d49-2n6gz"]
Oct 30 21:03:34.644569 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:34.644484    4521 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-c4b379-779cc77c6-b8gqg"]
Oct 30 21:03:34.663436 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:34.663399    4521 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-c4b379-7c594b8495-t7ncb"]
Oct 30 21:03:35.494401 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:35.494366    4521 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5fefb0b-nwgfc"]
Oct 30 21:03:35.494804 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:35.494787    4521 cpu_manager.go:414] "containerMap: removing container" podUID="9cb1b1d5-a595-403d-9a53-0560efac6ab7" containerName="istio-proxy"
Oct 30 21:03:35.494889 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:35.494806    4521 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9cb1b1d5-a595-403d-9a53-0560efac6ab7" containerName="istio-proxy"
Oct 30 21:03:35.494889 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:35.494823    4521 cpu_manager.go:414] "containerMap: removing container" podUID="d91a4f29-950e-4370-92e0-8ec493b72620" containerName="istio-proxy"
Oct 30 21:03:35.494889 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:35.494830    4521 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d91a4f29-950e-4370-92e0-8ec493b72620" containerName="istio-proxy"
Oct 30 21:03:35.494889 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:35.494845    4521 cpu_manager.go:414] "containerMap: removing container" podUID="11b85d0f-fa82-4861-9c1c-e47a1c1c308a" containerName="istio-proxy"
Oct 30 21:03:35.494889 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:35.494850    4521 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11b85d0f-fa82-4861-9c1c-e47a1c1c308a" containerName="istio-proxy"
Oct 30 21:03:35.494889 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:35.494861    4521 cpu_manager.go:414] "containerMap: removing container" podUID="7a5f91bb-7229-454f-a006-0d768f8879f4" containerName="istio-proxy"
Oct 30 21:03:35.494889 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:35.494866    4521 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a5f91bb-7229-454f-a006-0d768f8879f4" containerName="istio-proxy"
Oct 30 21:03:35.494889 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:35.494873    4521 cpu_manager.go:414] "containerMap: removing container" podUID="308f74fc-a395-4ad2-8902-8ed077ab9d0d" containerName="istio-proxy"
Oct 30 21:03:35.494889 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:35.494878    4521 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="308f74fc-a395-4ad2-8902-8ed077ab9d0d" containerName="istio-proxy"
Oct 30 21:03:35.494889 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:35.494886    4521 cpu_manager.go:414] "containerMap: removing container" podUID="6cff2b61-dcbd-4450-b776-43eff57a4fd2" containerName="istio-proxy"
Oct 30 21:03:35.494889 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:35.494891    4521 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6cff2b61-dcbd-4450-b776-43eff57a4fd2" containerName="istio-proxy"
Oct 30 21:03:35.495234 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:35.494898    4521 cpu_manager.go:414] "containerMap: removing container" podUID="d3990d1c-906f-4267-b67b-5b1309f20cfc" containerName="istio-proxy"
Oct 30 21:03:35.495234 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:35.494903    4521 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3990d1c-906f-4267-b67b-5b1309f20cfc" containerName="istio-proxy"
Oct 30 21:03:35.495234 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:35.494909    4521 cpu_manager.go:414] "containerMap: removing container" podUID="795d7e19-cf6e-4863-bf80-28ae03af6a98" containerName="istio-proxy"
Oct 30 21:03:35.495234 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:35.494914    4521 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="795d7e19-cf6e-4863-bf80-28ae03af6a98" containerName="istio-proxy"
Oct 30 21:03:35.495234 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:35.494922    4521 cpu_manager.go:414] "containerMap: removing container" podUID="75d8f639-8cee-4be3-af28-eb7886627983" containerName="revoke-keys"
Oct 30 21:03:35.495234 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:35.494927    4521 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75d8f639-8cee-4be3-af28-eb7886627983" containerName="revoke-keys"
Oct 30 21:03:35.495234 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:35.494935    4521 cpu_manager.go:414] "containerMap: removing container" podUID="b7f18233-f19c-4f74-a071-62d63402691c" containerName="istio-proxy"
Oct 30 21:03:35.495234 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:35.494940    4521 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7f18233-f19c-4f74-a071-62d63402691c" containerName="istio-proxy"
Oct 30 21:03:35.495234 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:35.494948    4521 cpu_manager.go:414] "containerMap: removing container" podUID="7d10d07e-a60e-4c6a-b067-66234939fa57" containerName="revoke-keys"
Oct 30 21:03:35.495234 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:35.494952    4521 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d10d07e-a60e-4c6a-b067-66234939fa57" containerName="revoke-keys"
Oct 30 21:03:35.495234 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:35.494957    4521 cpu_manager.go:414] "containerMap: removing container" podUID="65e4dd7b-506f-4350-a82c-e35e61706595" containerName="istio-proxy"
Oct 30 21:03:35.495234 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:35.494961    4521 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65e4dd7b-506f-4350-a82c-e35e61706595" containerName="istio-proxy"
Oct 30 21:03:35.495234 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:35.494969    4521 cpu_manager.go:414] "containerMap: removing container" podUID="9dc1356e-e12d-49f9-822e-f1ccf4fdaa3b" containerName="istio-proxy"
Oct 30 21:03:35.495234 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:35.494973    4521 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9dc1356e-e12d-49f9-822e-f1ccf4fdaa3b" containerName="istio-proxy"
Oct 30 21:03:35.495234 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:35.494980    4521 cpu_manager.go:414] "containerMap: removing container" podUID="69a73fe6-ec07-446c-aecc-e11806b5f492" containerName="istio-proxy"
Oct 30 21:03:35.495234 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:35.494984    4521 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69a73fe6-ec07-446c-aecc-e11806b5f492" containerName="istio-proxy"
Oct 30 21:03:35.495234 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:35.494994    4521 cpu_manager.go:414] "containerMap: removing container" podUID="7cb73975-c1ac-4957-9337-6ada4d05a2dc" containerName="maas-api"
Oct 30 21:03:35.495234 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:35.494999    4521 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7cb73975-c1ac-4957-9337-6ada4d05a2dc" containerName="maas-api"
Oct 30 21:03:35.495234 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:35.495005    4521 cpu_manager.go:414] "containerMap: removing container" podUID="1e857e76-3d48-4e5b-a8b5-830cfac13f90" containerName="revoke-keys"
Oct 30 21:03:35.495234 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:35.495008    4521 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e857e76-3d48-4e5b-a8b5-830cfac13f90" containerName="revoke-keys"
Oct 30 21:03:35.495234 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:35.495014    4521 cpu_manager.go:414] "containerMap: removing container" podUID="f860c372-4493-4c25-b46a-a6d3af710390" containerName="istio-proxy"
Oct 30 21:03:35.495234 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:35.495019    4521 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f860c372-4493-4c25-b46a-a6d3af710390" containerName="istio-proxy"
Oct 30 21:03:35.495234 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:35.495089    4521 memory_manager.go:365] "RemoveStaleState removing state" podUID="65e4dd7b-506f-4350-a82c-e35e61706595" containerName="istio-proxy"
Oct 30 21:03:35.495234 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:35.495096    4521 memory_manager.go:365] "RemoveStaleState removing state" podUID="9dc1356e-e12d-49f9-822e-f1ccf4fdaa3b" containerName="istio-proxy"
Oct 30 21:03:35.495234 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:35.495103    4521 memory_manager.go:365] "RemoveStaleState removing state" podUID="69a73fe6-ec07-446c-aecc-e11806b5f492" containerName="istio-proxy"
Oct 30 21:03:35.495234 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:35.495109    4521 memory_manager.go:365] "RemoveStaleState removing state" podUID="7cb73975-c1ac-4957-9337-6ada4d05a2dc" containerName="maas-api"
Oct 30 21:03:35.495234 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:35.495114    4521 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e857e76-3d48-4e5b-a8b5-830cfac13f90" containerName="revoke-keys"
Oct 30 21:03:35.495234 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:35.495119    4521 memory_manager.go:365] "RemoveStaleState removing state" podUID="f860c372-4493-4c25-b46a-a6d3af710390" containerName="istio-proxy"
Oct 30 21:03:35.495234 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:35.495126    4521 memory_manager.go:365] "RemoveStaleState removing state" podUID="9cb1b1d5-a595-403d-9a53-0560efac6ab7" containerName="istio-proxy"
Oct 30 21:03:35.495234 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:35.495131    4521 memory_manager.go:365] "RemoveStaleState removing state" podUID="11b85d0f-fa82-4861-9c1c-e47a1c1c308a" containerName="istio-proxy"
Oct 30 21:03:35.495234 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:35.495137    4521 memory_manager.go:365] "RemoveStaleState removing state" podUID="d91a4f29-950e-4370-92e0-8ec493b72620" containerName="istio-proxy"
Oct 30 21:03:35.495234 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:35.495143    4521 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a5f91bb-7229-454f-a006-0d768f8879f4" containerName="istio-proxy"
Oct 30 21:03:35.495234 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:35.495148    4521 memory_manager.go:365] "RemoveStaleState removing state" podUID="308f74fc-a395-4ad2-8902-8ed077ab9d0d" containerName="istio-proxy"
Oct 30 21:03:35.495234 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:35.495153    4521 memory_manager.go:365] "RemoveStaleState removing state" podUID="6cff2b61-dcbd-4450-b776-43eff57a4fd2" containerName="istio-proxy"
Oct 30 21:03:35.495234 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:35.495158    4521 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3990d1c-906f-4267-b67b-5b1309f20cfc" containerName="istio-proxy"
Oct 30 21:03:35.495234 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:35.495163    4521 memory_manager.go:365] "RemoveStaleState removing state" podUID="795d7e19-cf6e-4863-bf80-28ae03af6a98" containerName="istio-proxy"
Oct 30 21:03:35.495234 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:35.495169    4521 memory_manager.go:365] "RemoveStaleState removing state" podUID="75d8f639-8cee-4be3-af28-eb7886627983" containerName="revoke-keys"
Oct 30 21:03:35.495234 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:35.495174    4521 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7f18233-f19c-4f74-a071-62d63402691c" containerName="istio-proxy"
Oct 30 21:03:35.495234 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:35.495179    4521 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d10d07e-a60e-4c6a-b067-66234939fa57" containerName="revoke-keys"
Oct 30 21:03:35.498996 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:35.498980    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5fefb0b-nwgfc"
Oct 30 21:03:35.513063 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:35.513033    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5fefb0b-nwgfc"]
Oct 30 21:03:35.525187 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:35.525146    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/584d633e-1e52-4eb5-a159-e64d2b9d37e7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b5fefb0b-nwgfc\" (UID: \"584d633e-1e52-4eb5-a159-e64d2b9d37e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5fefb0b-nwgfc"
Oct 30 21:03:35.525364 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:35.525200    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/584d633e-1e52-4eb5-a159-e64d2b9d37e7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b5fefb0b-nwgfc\" (UID: \"584d633e-1e52-4eb5-a159-e64d2b9d37e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5fefb0b-nwgfc"
Oct 30 21:03:35.525420 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:35.525393    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvpgd\" (UniqueName: \"kubernetes.io/projected/584d633e-1e52-4eb5-a159-e64d2b9d37e7-kube-api-access-xvpgd\") pod \"maas-api-revoke-sub-b5fefb0b-nwgfc\" (UID: \"584d633e-1e52-4eb5-a159-e64d2b9d37e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5fefb0b-nwgfc"
Oct 30 21:03:35.583666 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:35.583624    4521 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:03:35.583666 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:35.583667    4521 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:03:35.583893 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:35.583777    4521 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c45pw\""
Oct 30 21:03:35.583893 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:35.583840    4521 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:03:35.626632 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:35.626599    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvpgd\" (UniqueName: \"kubernetes.io/projected/584d633e-1e52-4eb5-a159-e64d2b9d37e7-kube-api-access-xvpgd\") pod \"maas-api-revoke-sub-b5fefb0b-nwgfc\" (UID: \"584d633e-1e52-4eb5-a159-e64d2b9d37e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5fefb0b-nwgfc"
Oct 30 21:03:35.626875 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:35.626647    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/584d633e-1e52-4eb5-a159-e64d2b9d37e7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b5fefb0b-nwgfc\" (UID: \"584d633e-1e52-4eb5-a159-e64d2b9d37e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5fefb0b-nwgfc"
Oct 30 21:03:35.626875 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:35.626677    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/584d633e-1e52-4eb5-a159-e64d2b9d37e7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b5fefb0b-nwgfc\" (UID: \"584d633e-1e52-4eb5-a159-e64d2b9d37e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5fefb0b-nwgfc"
Oct 30 21:03:35.627347 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:35.627304    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/584d633e-1e52-4eb5-a159-e64d2b9d37e7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b5fefb0b-nwgfc\" (UID: \"584d633e-1e52-4eb5-a159-e64d2b9d37e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5fefb0b-nwgfc"
Oct 30 21:03:35.629036 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:35.629015    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/584d633e-1e52-4eb5-a159-e64d2b9d37e7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b5fefb0b-nwgfc\" (UID: \"584d633e-1e52-4eb5-a159-e64d2b9d37e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5fefb0b-nwgfc"
Oct 30 21:03:35.639823 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:35.639796    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvpgd\" (UniqueName: \"kubernetes.io/projected/584d633e-1e52-4eb5-a159-e64d2b9d37e7-kube-api-access-xvpgd\") pod \"maas-api-revoke-sub-b5fefb0b-nwgfc\" (UID: \"584d633e-1e52-4eb5-a159-e64d2b9d37e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5fefb0b-nwgfc"
Oct 30 21:03:35.813914 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:35.813833    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5fefb0b-nwgfc"
Oct 30 21:03:36.098373 ip-10-0-131-125 kubenswrapper[4521]: W1030 21:03:36.098339    4521 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod584d633e_1e52_4eb5_a159_e64d2b9d37e7.slice/crio-4899d527dfc7ebdd9592eeea50406b8a4e59bbf7d35eb9e4959cc329229db8f3 WatchSource:0}: Error finding container 4899d527dfc7ebdd9592eeea50406b8a4e59bbf7d35eb9e4959cc329229db8f3: Status 404 returned error can't find the container with id 4899d527dfc7ebdd9592eeea50406b8a4e59bbf7d35eb9e4959cc329229db8f3
Oct 30 21:03:36.098611 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:36.098589    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5fefb0b-nwgfc"]
Oct 30 21:03:36.622413 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:36.622369    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5fefb0b-nwgfc" event={"ID":"584d633e-1e52-4eb5-a159-e64d2b9d37e7","Type":"ContainerStarted","Data":"ec55a4368f21b5a8515dc4b6387e36f645a1105661ae41d3f2ae7d16d642c0c1"}
Oct 30 21:03:36.622601 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:36.622457    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5fefb0b-nwgfc" event={"ID":"584d633e-1e52-4eb5-a159-e64d2b9d37e7","Type":"ContainerStarted","Data":"4899d527dfc7ebdd9592eeea50406b8a4e59bbf7d35eb9e4959cc329229db8f3"}
Oct 30 21:03:36.650504 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:36.650451    4521 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5fefb0b-nwgfc" podStartSLOduration=1.65043757 podStartE2EDuration="1.65043757s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:03:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:03:36.646317701 +0000 UTC m=+1483.991938940" watchObservedRunningTime="2026-10-30 21:03:36.65043757 +0000 UTC m=+1483.996058810"
Oct 30 21:03:37.633173 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:37.633140    4521 generic.go:356] "Generic (PLEG): container finished" podID="584d633e-1e52-4eb5-a159-e64d2b9d37e7" containerID="ec55a4368f21b5a8515dc4b6387e36f645a1105661ae41d3f2ae7d16d642c0c1" exitCode=0
Oct 30 21:03:37.633648 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:37.633223    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5fefb0b-nwgfc" event={"ID":"584d633e-1e52-4eb5-a159-e64d2b9d37e7","Type":"ContainerDied","Data":"ec55a4368f21b5a8515dc4b6387e36f645a1105661ae41d3f2ae7d16d642c0c1"}
Oct 30 21:03:38.784856 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:38.784826    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5fefb0b-nwgfc"
Oct 30 21:03:38.861273 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:38.861233    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/584d633e-1e52-4eb5-a159-e64d2b9d37e7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/584d633e-1e52-4eb5-a159-e64d2b9d37e7-maas-api-xks-ca\") pod \"584d633e-1e52-4eb5-a159-e64d2b9d37e7\" (UID: \"584d633e-1e52-4eb5-a159-e64d2b9d37e7\") "
Oct 30 21:03:38.861273 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:38.861272    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/584d633e-1e52-4eb5-a159-e64d2b9d37e7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/584d633e-1e52-4eb5-a159-e64d2b9d37e7-maas-api-openshift-ca\") pod \"584d633e-1e52-4eb5-a159-e64d2b9d37e7\" (UID: \"584d633e-1e52-4eb5-a159-e64d2b9d37e7\") "
Oct 30 21:03:38.861549 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:38.861372    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/584d633e-1e52-4eb5-a159-e64d2b9d37e7-kube-api-access-xvpgd\" (UniqueName: \"kubernetes.io/projected/584d633e-1e52-4eb5-a159-e64d2b9d37e7-kube-api-access-xvpgd\") pod \"584d633e-1e52-4eb5-a159-e64d2b9d37e7\" (UID: \"584d633e-1e52-4eb5-a159-e64d2b9d37e7\") "
Oct 30 21:03:38.861648 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:38.861624    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/584d633e-1e52-4eb5-a159-e64d2b9d37e7-maas-api-openshift-ca" pod "584d633e-1e52-4eb5-a159-e64d2b9d37e7" (UID: "584d633e-1e52-4eb5-a159-e64d2b9d37e7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:03:38.861772 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:38.861754    4521 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/584d633e-1e52-4eb5-a159-e64d2b9d37e7-maas-api-openshift-ca\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:03:38.863490 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:38.863467    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/584d633e-1e52-4eb5-a159-e64d2b9d37e7-maas-api-xks-ca" pod "584d633e-1e52-4eb5-a159-e64d2b9d37e7" (UID: "584d633e-1e52-4eb5-a159-e64d2b9d37e7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:03:38.863490 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:38.863472    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/584d633e-1e52-4eb5-a159-e64d2b9d37e7-kube-api-access-xvpgd" pod "584d633e-1e52-4eb5-a159-e64d2b9d37e7" (UID: "584d633e-1e52-4eb5-a159-e64d2b9d37e7"). InnerVolumeSpecName "kube-api-access-xvpgd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:03:38.963240 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:38.963206    4521 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvpgd\" (UniqueName: \"kubernetes.io/projected/584d633e-1e52-4eb5-a159-e64d2b9d37e7-kube-api-access-xvpgd\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:03:38.963240 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:38.963234    4521 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/584d633e-1e52-4eb5-a159-e64d2b9d37e7-maas-api-xks-ca\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:03:39.645618 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:39.645571    4521 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-c4b379-779cc77c6-b8gqg" podUID="c29ead58-26c1-4720-ac49-99b5e045b85d" containerName="payload-processing" containerID="cri-o://9ff6b4c9dad5e2f78b264f46d6a0af55d026adc75fb3ec343f9e03c2eb415d3e" gracePeriod=25
Oct 30 21:03:39.649119 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:39.649095    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5fefb0b-nwgfc"
Oct 30 21:03:39.649232 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:39.649132    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5fefb0b-nwgfc" event={"ID":"584d633e-1e52-4eb5-a159-e64d2b9d37e7","Type":"ContainerDied","Data":"4899d527dfc7ebdd9592eeea50406b8a4e59bbf7d35eb9e4959cc329229db8f3"}
Oct 30 21:03:39.649232 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:39.649191    4521 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4899d527dfc7ebdd9592eeea50406b8a4e59bbf7d35eb9e4959cc329229db8f3"
Oct 30 21:03:39.664488 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:39.664458    4521 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c4b379-7c594b8495-t7ncb" podUID="77f3a44b-d4f2-4506-b386-7a216ee118d9" containerName="payload-pre-processing" containerID="cri-o://9fb1f2ff6267618504e29c1f8c48c620c26f736f88391c462f6c660047b457e1" gracePeriod=25
Oct 30 21:03:44.420510 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:44.420454    4521 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-c4b379/distinct-2-w1-c4b379-kserve-7999b785f5-97fz5" podUID="080a654d-a5bf-4845-a07f-cc8a7e6d0ace" containerName="main" containerID="cri-o://89bfd3d479cd779cb5afde54eac0acbb025cec7a5720c730ff0aa0ea1a55c2f0" gracePeriod=45
Oct 30 21:03:44.440129 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:44.440072    4521 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-c4b379/premium-w1-c4b379-kserve-575f896d49-2n6gz" podUID="2aac3e8e-d631-4b9c-bdbb-3b4b1df133a9" containerName="main" containerID="cri-o://58c11395b2f1ed819b5cc7f382bc5c969e7a03e693cd49102fcf67ea827d642c" gracePeriod=45
Oct 30 21:03:44.693349 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:44.693284    4521 generic.go:356] "Generic (PLEG): container finished" podID="2aac3e8e-d631-4b9c-bdbb-3b4b1df133a9" containerID="58c11395b2f1ed819b5cc7f382bc5c969e7a03e693cd49102fcf67ea827d642c" exitCode=0
Oct 30 21:03:44.693583 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:44.693548    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-c4b379/premium-w1-c4b379-kserve-575f896d49-2n6gz" event={"ID":"2aac3e8e-d631-4b9c-bdbb-3b4b1df133a9","Type":"ContainerDied","Data":"58c11395b2f1ed819b5cc7f382bc5c969e7a03e693cd49102fcf67ea827d642c"}
Oct 30 21:03:44.695134 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:44.695091    4521 generic.go:356] "Generic (PLEG): container finished" podID="080a654d-a5bf-4845-a07f-cc8a7e6d0ace" containerID="89bfd3d479cd779cb5afde54eac0acbb025cec7a5720c730ff0aa0ea1a55c2f0" exitCode=0
Oct 30 21:03:44.695224 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:44.695166    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-c4b379/distinct-2-w1-c4b379-kserve-7999b785f5-97fz5" event={"ID":"080a654d-a5bf-4845-a07f-cc8a7e6d0ace","Type":"ContainerDied","Data":"89bfd3d479cd779cb5afde54eac0acbb025cec7a5720c730ff0aa0ea1a55c2f0"}
Oct 30 21:03:44.707418 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:44.707386    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-c4b379/distinct-2-w1-c4b379-kserve-7999b785f5-97fz5"
Oct 30 21:03:44.718609 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:44.718582    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-c4b379/premium-w1-c4b379-kserve-575f896d49-2n6gz"
Oct 30 21:03:44.829287 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:44.829253    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/080a654d-a5bf-4845-a07f-cc8a7e6d0ace-dshm\" (UniqueName: \"kubernetes.io/empty-dir/080a654d-a5bf-4845-a07f-cc8a7e6d0ace-dshm\") pod \"080a654d-a5bf-4845-a07f-cc8a7e6d0ace\" (UID: \"080a654d-a5bf-4845-a07f-cc8a7e6d0ace\") "
Oct 30 21:03:44.829494 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:44.829309    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2aac3e8e-d631-4b9c-bdbb-3b4b1df133a9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2aac3e8e-d631-4b9c-bdbb-3b4b1df133a9-tmp-dir\") pod \"2aac3e8e-d631-4b9c-bdbb-3b4b1df133a9\" (UID: \"2aac3e8e-d631-4b9c-bdbb-3b4b1df133a9\") "
Oct 30 21:03:44.829494 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:44.829361    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/080a654d-a5bf-4845-a07f-cc8a7e6d0ace-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/080a654d-a5bf-4845-a07f-cc8a7e6d0ace-model-cache\") pod \"080a654d-a5bf-4845-a07f-cc8a7e6d0ace\" (UID: \"080a654d-a5bf-4845-a07f-cc8a7e6d0ace\") "
Oct 30 21:03:44.829494 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:44.829384    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/080a654d-a5bf-4845-a07f-cc8a7e6d0ace-kube-api-access-zw5t2\" (UniqueName: \"kubernetes.io/projected/080a654d-a5bf-4845-a07f-cc8a7e6d0ace-kube-api-access-zw5t2\") pod \"080a654d-a5bf-4845-a07f-cc8a7e6d0ace\" (UID: \"080a654d-a5bf-4845-a07f-cc8a7e6d0ace\") "
Oct 30 21:03:44.829494 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:44.829437    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2aac3e8e-d631-4b9c-bdbb-3b4b1df133a9-home\" (UniqueName: \"kubernetes.io/empty-dir/2aac3e8e-d631-4b9c-bdbb-3b4b1df133a9-home\") pod \"2aac3e8e-d631-4b9c-bdbb-3b4b1df133a9\" (UID: \"2aac3e8e-d631-4b9c-bdbb-3b4b1df133a9\") "
Oct 30 21:03:44.829494 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:44.829478    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/080a654d-a5bf-4845-a07f-cc8a7e6d0ace-tls-certs\" (UniqueName: \"kubernetes.io/secret/080a654d-a5bf-4845-a07f-cc8a7e6d0ace-tls-certs\") pod \"080a654d-a5bf-4845-a07f-cc8a7e6d0ace\" (UID: \"080a654d-a5bf-4845-a07f-cc8a7e6d0ace\") "
Oct 30 21:03:44.829837 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:44.829521    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2aac3e8e-d631-4b9c-bdbb-3b4b1df133a9-tls-certs\" (UniqueName: \"kubernetes.io/secret/2aac3e8e-d631-4b9c-bdbb-3b4b1df133a9-tls-certs\") pod \"2aac3e8e-d631-4b9c-bdbb-3b4b1df133a9\" (UID: \"2aac3e8e-d631-4b9c-bdbb-3b4b1df133a9\") "
Oct 30 21:03:44.829837 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:44.829570    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/080a654d-a5bf-4845-a07f-cc8a7e6d0ace-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/080a654d-a5bf-4845-a07f-cc8a7e6d0ace-tmp-dir\") pod \"080a654d-a5bf-4845-a07f-cc8a7e6d0ace\" (UID: \"080a654d-a5bf-4845-a07f-cc8a7e6d0ace\") "
Oct 30 21:03:44.829837 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:44.829621    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2aac3e8e-d631-4b9c-bdbb-3b4b1df133a9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2aac3e8e-d631-4b9c-bdbb-3b4b1df133a9-model-cache\") pod \"2aac3e8e-d631-4b9c-bdbb-3b4b1df133a9\" (UID: \"2aac3e8e-d631-4b9c-bdbb-3b4b1df133a9\") "
Oct 30 21:03:44.829837 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:44.829670    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2aac3e8e-d631-4b9c-bdbb-3b4b1df133a9-kube-api-access-7lrj6\" (UniqueName: \"kubernetes.io/projected/2aac3e8e-d631-4b9c-bdbb-3b4b1df133a9-kube-api-access-7lrj6\") pod \"2aac3e8e-d631-4b9c-bdbb-3b4b1df133a9\" (UID: \"2aac3e8e-d631-4b9c-bdbb-3b4b1df133a9\") "
Oct 30 21:03:44.829837 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:44.829705    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/080a654d-a5bf-4845-a07f-cc8a7e6d0ace-home\" (UniqueName: \"kubernetes.io/empty-dir/080a654d-a5bf-4845-a07f-cc8a7e6d0ace-home\") pod \"080a654d-a5bf-4845-a07f-cc8a7e6d0ace\" (UID: \"080a654d-a5bf-4845-a07f-cc8a7e6d0ace\") "
Oct 30 21:03:44.829837 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:44.829732    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2aac3e8e-d631-4b9c-bdbb-3b4b1df133a9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2aac3e8e-d631-4b9c-bdbb-3b4b1df133a9-dshm\") pod \"2aac3e8e-d631-4b9c-bdbb-3b4b1df133a9\" (UID: \"2aac3e8e-d631-4b9c-bdbb-3b4b1df133a9\") "
Oct 30 21:03:44.830142 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:44.829566    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2aac3e8e-d631-4b9c-bdbb-3b4b1df133a9-tmp-dir" pod "2aac3e8e-d631-4b9c-bdbb-3b4b1df133a9" (UID: "2aac3e8e-d631-4b9c-bdbb-3b4b1df133a9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:03:44.830142 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:44.830019    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/080a654d-a5bf-4845-a07f-cc8a7e6d0ace-home" pod "080a654d-a5bf-4845-a07f-cc8a7e6d0ace" (UID: "080a654d-a5bf-4845-a07f-cc8a7e6d0ace"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:03:44.830142 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:44.829626    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/080a654d-a5bf-4845-a07f-cc8a7e6d0ace-model-cache" pod "080a654d-a5bf-4845-a07f-cc8a7e6d0ace" (UID: "080a654d-a5bf-4845-a07f-cc8a7e6d0ace"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:03:44.830142 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:44.829759    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2aac3e8e-d631-4b9c-bdbb-3b4b1df133a9-home" pod "2aac3e8e-d631-4b9c-bdbb-3b4b1df133a9" (UID: "2aac3e8e-d631-4b9c-bdbb-3b4b1df133a9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:03:44.830142 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:44.829822    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/080a654d-a5bf-4845-a07f-cc8a7e6d0ace-tmp-dir" pod "080a654d-a5bf-4845-a07f-cc8a7e6d0ace" (UID: "080a654d-a5bf-4845-a07f-cc8a7e6d0ace"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:03:44.830142 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:44.829886    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2aac3e8e-d631-4b9c-bdbb-3b4b1df133a9-model-cache" pod "2aac3e8e-d631-4b9c-bdbb-3b4b1df133a9" (UID: "2aac3e8e-d631-4b9c-bdbb-3b4b1df133a9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:03:44.830449 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:44.830395    4521 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/080a654d-a5bf-4845-a07f-cc8a7e6d0ace-tmp-dir\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:03:44.830449 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:44.830418    4521 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2aac3e8e-d631-4b9c-bdbb-3b4b1df133a9-model-cache\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:03:44.830449 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:44.830436    4521 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/080a654d-a5bf-4845-a07f-cc8a7e6d0ace-home\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:03:44.830613 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:44.830450    4521 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2aac3e8e-d631-4b9c-bdbb-3b4b1df133a9-tmp-dir\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:03:44.830613 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:44.830462    4521 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/080a654d-a5bf-4845-a07f-cc8a7e6d0ace-model-cache\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:03:44.830613 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:44.830476    4521 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2aac3e8e-d631-4b9c-bdbb-3b4b1df133a9-home\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:03:44.832004 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:44.831974    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/080a654d-a5bf-4845-a07f-cc8a7e6d0ace-tls-certs" pod "080a654d-a5bf-4845-a07f-cc8a7e6d0ace" (UID: "080a654d-a5bf-4845-a07f-cc8a7e6d0ace"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:03:44.832004 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:44.831981    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/080a654d-a5bf-4845-a07f-cc8a7e6d0ace-dshm" pod "080a654d-a5bf-4845-a07f-cc8a7e6d0ace" (UID: "080a654d-a5bf-4845-a07f-cc8a7e6d0ace"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:03:44.832143 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:44.832073    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2aac3e8e-d631-4b9c-bdbb-3b4b1df133a9-tls-certs" pod "2aac3e8e-d631-4b9c-bdbb-3b4b1df133a9" (UID: "2aac3e8e-d631-4b9c-bdbb-3b4b1df133a9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:03:44.832143 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:44.832087    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/080a654d-a5bf-4845-a07f-cc8a7e6d0ace-kube-api-access-zw5t2" pod "080a654d-a5bf-4845-a07f-cc8a7e6d0ace" (UID: "080a654d-a5bf-4845-a07f-cc8a7e6d0ace"). InnerVolumeSpecName "kube-api-access-zw5t2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:03:44.832261 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:44.832245    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2aac3e8e-d631-4b9c-bdbb-3b4b1df133a9-dshm" pod "2aac3e8e-d631-4b9c-bdbb-3b4b1df133a9" (UID: "2aac3e8e-d631-4b9c-bdbb-3b4b1df133a9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:03:44.832313 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:44.832259    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2aac3e8e-d631-4b9c-bdbb-3b4b1df133a9-kube-api-access-7lrj6" pod "2aac3e8e-d631-4b9c-bdbb-3b4b1df133a9" (UID: "2aac3e8e-d631-4b9c-bdbb-3b4b1df133a9"). InnerVolumeSpecName "kube-api-access-7lrj6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:03:44.931415 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:44.931367    4521 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2aac3e8e-d631-4b9c-bdbb-3b4b1df133a9-tls-certs\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:03:44.931415 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:44.931408    4521 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7lrj6\" (UniqueName: \"kubernetes.io/projected/2aac3e8e-d631-4b9c-bdbb-3b4b1df133a9-kube-api-access-7lrj6\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:03:44.931415 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:44.931421    4521 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2aac3e8e-d631-4b9c-bdbb-3b4b1df133a9-dshm\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:03:44.931677 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:44.931433    4521 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/080a654d-a5bf-4845-a07f-cc8a7e6d0ace-dshm\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:03:44.931677 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:44.931445    4521 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zw5t2\" (UniqueName: \"kubernetes.io/projected/080a654d-a5bf-4845-a07f-cc8a7e6d0ace-kube-api-access-zw5t2\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:03:44.931677 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:44.931456    4521 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/080a654d-a5bf-4845-a07f-cc8a7e6d0ace-tls-certs\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:03:45.704354 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:45.704310    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-c4b379/premium-w1-c4b379-kserve-575f896d49-2n6gz"
Oct 30 21:03:45.704850 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:45.704350    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-c4b379/premium-w1-c4b379-kserve-575f896d49-2n6gz" event={"ID":"2aac3e8e-d631-4b9c-bdbb-3b4b1df133a9","Type":"ContainerDied","Data":"f1dbafc2a63346816cf84a5612f0abb7d1e3d794c46fdb48d7c5760f638bf3c9"}
Oct 30 21:03:45.704850 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:45.704412    4521 scope.go:122] "RemoveContainer" containerID="58c11395b2f1ed819b5cc7f382bc5c969e7a03e693cd49102fcf67ea827d642c"
Oct 30 21:03:45.706592 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:45.706569    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-c4b379/distinct-2-w1-c4b379-kserve-7999b785f5-97fz5" event={"ID":"080a654d-a5bf-4845-a07f-cc8a7e6d0ace","Type":"ContainerDied","Data":"f2e2e02d26328846b91e36984c550935a9a110628cf8a7520351633ac247fcb4"}
Oct 30 21:03:45.706723 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:45.706607    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-c4b379/distinct-2-w1-c4b379-kserve-7999b785f5-97fz5"
Oct 30 21:03:45.717153 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:45.717130    4521 scope.go:122] "RemoveContainer" containerID="89bfd3d479cd779cb5afde54eac0acbb025cec7a5720c730ff0aa0ea1a55c2f0"
Oct 30 21:03:45.734067 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:45.734039    4521 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-c4b379/premium-w1-c4b379-kserve-575f896d49-2n6gz"]
Oct 30 21:03:45.741295 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:45.741268    4521 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-c4b379/premium-w1-c4b379-kserve-575f896d49-2n6gz"]
Oct 30 21:03:45.753730 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:45.753697    4521 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-c4b379/distinct-2-w1-c4b379-kserve-7999b785f5-97fz5"]
Oct 30 21:03:45.757353 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:45.757317    4521 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-c4b379/distinct-2-w1-c4b379-kserve-7999b785f5-97fz5"]
Oct 30 21:03:46.428426 ip-10-0-131-125 kubenswrapper[4521]: E1030 21:03:46.428377    4521 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lkn2k" containerName="extractor"
Oct 30 21:03:46.796650 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:46.793991    4521 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="080a654d-a5bf-4845-a07f-cc8a7e6d0ace" path="/var/lib/kubelet/pods/080a654d-a5bf-4845-a07f-cc8a7e6d0ace/volumes"
Oct 30 21:03:46.796650 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:46.794735    4521 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2aac3e8e-d631-4b9c-bdbb-3b4b1df133a9" path="/var/lib/kubelet/pods/2aac3e8e-d631-4b9c-bdbb-3b4b1df133a9/volumes"
Oct 30 21:03:47.744319 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:47.744271    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4kxb"]
Oct 30 21:03:48.596315 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:48.596271    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c4b379-7c594b8495-t7ncb" podUID="77f3a44b-d4f2-4506-b386-7a216ee118d9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:03:48.596315 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:48.596301    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-c4b379-779cc77c6-b8gqg" podUID="c29ead58-26c1-4720-ac49-99b5e045b85d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:03:53.916618 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:53.916585    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4kxb"]
Oct 30 21:03:57.651606 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:57.651571    4521 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-7mf77"]
Oct 30 21:03:57.652220 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:57.651826    4521 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-7mf77" podUID="2f940874-3c9d-4de3-985d-5d3c69b0edd3" containerName="istio-proxy" containerID="cri-o://4b4499114c447412b06dd2081f5c6ce55e0795bb55f022f9d305b6f629db1afb" gracePeriod=30
Oct 30 21:03:57.659389 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:57.659360    4521 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-lp6sk"]
Oct 30 21:03:57.659653 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:57.659607    4521 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-lp6sk" podUID="fbdd2291-639e-4c1e-986d-425c4ab63dbc" containerName="istio-proxy" containerID="cri-o://026db3fbe3f5478b701e10343c4817d9fe47d7df58e5baf741b24758eb686213" gracePeriod=30
Oct 30 21:03:57.665403 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:57.665369    4521 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-hz8mp"]
Oct 30 21:03:57.665618 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:57.665594    4521 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-hz8mp" podUID="5b77d06a-b7a8-4b2d-a365-a728d4397d8b" containerName="istio-proxy" containerID="cri-o://1fc06542506386cd87be1aa2f62f7c628d987268e35a2f5c5f557c0249431354" gracePeriod=30
Oct 30 21:03:58.595760 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:58.595717    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c4b379-7c594b8495-t7ncb" podUID="77f3a44b-d4f2-4506-b386-7a216ee118d9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:03:58.595940 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:58.595722    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-c4b379-779cc77c6-b8gqg" podUID="c29ead58-26c1-4720-ac49-99b5e045b85d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:03:59.868020 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:59.867749    4521 generic.go:356] "Generic (PLEG): container finished" podID="c29ead58-26c1-4720-ac49-99b5e045b85d" containerID="9ff6b4c9dad5e2f78b264f46d6a0af55d026adc75fb3ec343f9e03c2eb415d3e" exitCode=0
Oct 30 21:03:59.868020 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:59.867944    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-c4b379-779cc77c6-b8gqg" event={"ID":"c29ead58-26c1-4720-ac49-99b5e045b85d","Type":"ContainerDied","Data":"9ff6b4c9dad5e2f78b264f46d6a0af55d026adc75fb3ec343f9e03c2eb415d3e"}
Oct 30 21:03:59.870910 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:59.870742    4521 generic.go:356] "Generic (PLEG): container finished" podID="77f3a44b-d4f2-4506-b386-7a216ee118d9" containerID="9fb1f2ff6267618504e29c1f8c48c620c26f736f88391c462f6c660047b457e1" exitCode=0
Oct 30 21:03:59.870910 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:59.870816    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c4b379-7c594b8495-t7ncb" event={"ID":"77f3a44b-d4f2-4506-b386-7a216ee118d9","Type":"ContainerDied","Data":"9fb1f2ff6267618504e29c1f8c48c620c26f736f88391c462f6c660047b457e1"}
Oct 30 21:03:59.923394 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:59.923365    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-c4b379-779cc77c6-b8gqg"
Oct 30 21:03:59.955193 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:03:59.955164    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c4b379-7c594b8495-t7ncb"
Oct 30 21:04:00.007076 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:04:00.007039    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c29ead58-26c1-4720-ac49-99b5e045b85d-kube-api-access-6b86c\" (UniqueName: \"kubernetes.io/projected/c29ead58-26c1-4720-ac49-99b5e045b85d-kube-api-access-6b86c\") pod \"c29ead58-26c1-4720-ac49-99b5e045b85d\" (UID: \"c29ead58-26c1-4720-ac49-99b5e045b85d\") "
Oct 30 21:04:00.007248 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:04:00.007153    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c29ead58-26c1-4720-ac49-99b5e045b85d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c29ead58-26c1-4720-ac49-99b5e045b85d-plugins-config-volume\") pod \"c29ead58-26c1-4720-ac49-99b5e045b85d\" (UID: \"c29ead58-26c1-4720-ac49-99b5e045b85d\") "
Oct 30 21:04:00.007641 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:04:00.007613    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c29ead58-26c1-4720-ac49-99b5e045b85d-plugins-config-volume" pod "c29ead58-26c1-4720-ac49-99b5e045b85d" (UID: "c29ead58-26c1-4720-ac49-99b5e045b85d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:04:00.009259 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:04:00.009233    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c29ead58-26c1-4720-ac49-99b5e045b85d-kube-api-access-6b86c" pod "c29ead58-26c1-4720-ac49-99b5e045b85d" (UID: "c29ead58-26c1-4720-ac49-99b5e045b85d"). InnerVolumeSpecName "kube-api-access-6b86c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:04:00.108036 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:04:00.108003    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77f3a44b-d4f2-4506-b386-7a216ee118d9-kube-api-access-qpwq5\" (UniqueName: \"kubernetes.io/projected/77f3a44b-d4f2-4506-b386-7a216ee118d9-kube-api-access-qpwq5\") pod \"77f3a44b-d4f2-4506-b386-7a216ee118d9\" (UID: \"77f3a44b-d4f2-4506-b386-7a216ee118d9\") "
Oct 30 21:04:00.108036 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:04:00.108039    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77f3a44b-d4f2-4506-b386-7a216ee118d9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77f3a44b-d4f2-4506-b386-7a216ee118d9-plugins-config-volume\") pod \"77f3a44b-d4f2-4506-b386-7a216ee118d9\" (UID: \"77f3a44b-d4f2-4506-b386-7a216ee118d9\") "
Oct 30 21:04:00.108399 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:04:00.108314    4521 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c29ead58-26c1-4720-ac49-99b5e045b85d-plugins-config-volume\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:04:00.108399 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:04:00.108372    4521 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6b86c\" (UniqueName: \"kubernetes.io/projected/c29ead58-26c1-4720-ac49-99b5e045b85d-kube-api-access-6b86c\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:04:00.108548 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:04:00.108493    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77f3a44b-d4f2-4506-b386-7a216ee118d9-plugins-config-volume" pod "77f3a44b-d4f2-4506-b386-7a216ee118d9" (UID: "77f3a44b-d4f2-4506-b386-7a216ee118d9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:04:00.110154 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:04:00.110133    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77f3a44b-d4f2-4506-b386-7a216ee118d9-kube-api-access-qpwq5" pod "77f3a44b-d4f2-4506-b386-7a216ee118d9" (UID: "77f3a44b-d4f2-4506-b386-7a216ee118d9"). InnerVolumeSpecName "kube-api-access-qpwq5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:04:00.209121 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:04:00.209089    4521 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qpwq5\" (UniqueName: \"kubernetes.io/projected/77f3a44b-d4f2-4506-b386-7a216ee118d9-kube-api-access-qpwq5\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:04:00.209121 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:04:00.209120    4521 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77f3a44b-d4f2-4506-b386-7a216ee118d9-plugins-config-volume\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:04:00.879223 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:04:00.879196    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-c4b379-779cc77c6-b8gqg"
Oct 30 21:04:00.879727 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:04:00.879232    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-c4b379-779cc77c6-b8gqg" event={"ID":"c29ead58-26c1-4720-ac49-99b5e045b85d","Type":"ContainerDied","Data":"b969b1211f7c59b209365e82d4808d23da178eb90b70162032497b573256213b"}
Oct 30 21:04:00.879727 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:04:00.879289    4521 scope.go:122] "RemoveContainer" containerID="9ff6b4c9dad5e2f78b264f46d6a0af55d026adc75fb3ec343f9e03c2eb415d3e"
Oct 30 21:04:00.880927 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:04:00.880908    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c4b379-7c594b8495-t7ncb"
Oct 30 21:04:00.881006 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:04:00.880933    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c4b379-7c594b8495-t7ncb" event={"ID":"77f3a44b-d4f2-4506-b386-7a216ee118d9","Type":"ContainerDied","Data":"a7fd6974ffd5be56409d8b5a3e664da90b6e2633a0b9e37d8b58b7ec8eab88a0"}
Oct 30 21:04:00.889878 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:04:00.889847    4521 scope.go:122] "RemoveContainer" containerID="9fb1f2ff6267618504e29c1f8c48c620c26f736f88391c462f6c660047b457e1"
Oct 30 21:04:00.902213 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:04:00.902181    4521 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-c4b379-7c594b8495-t7ncb"]
Oct 30 21:04:00.906246 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:04:00.906226    4521 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-c4b379-7c594b8495-t7ncb"]
Oct 30 21:04:00.917557 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:04:00.917495    4521 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-c4b379-779cc77c6-b8gqg"]
Oct 30 21:04:00.923133 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:04:00.923110    4521 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-c4b379-779cc77c6-b8gqg"]
Oct 30 21:04:02.902557 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:04:02.902453    4521 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77f3a44b-d4f2-4506-b386-7a216ee118d9" path="/var/lib/kubelet/pods/77f3a44b-d4f2-4506-b386-7a216ee118d9/volumes"
Oct 30 21:04:02.903922 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:04:02.903484    4521 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c29ead58-26c1-4720-ac49-99b5e045b85d" path="/var/lib/kubelet/pods/c29ead58-26c1-4720-ac49-99b5e045b85d/volumes"
Oct 30 21:04:02.967409 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:04:02.957100    4521 generic.go:356] "Generic (PLEG): container finished" podID="5b77d06a-b7a8-4b2d-a365-a728d4397d8b" containerID="1fc06542506386cd87be1aa2f62f7c628d987268e35a2f5c5f557c0249431354" exitCode=0
Oct 30 21:04:02.967409 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:04:02.957209    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-hz8mp" event={"ID":"5b77d06a-b7a8-4b2d-a365-a728d4397d8b","Type":"ContainerDied","Data":"1fc06542506386cd87be1aa2f62f7c628d987268e35a2f5c5f557c0249431354"}
Oct 30 21:04:02.968987 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:04:02.967762    4521 generic.go:356] "Generic (PLEG): container finished" podID="2f940874-3c9d-4de3-985d-5d3c69b0edd3" containerID="4b4499114c447412b06dd2081f5c6ce55e0795bb55f022f9d305b6f629db1afb" exitCode=0
Oct 30 21:04:02.968987 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:04:02.968540    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-7mf77" event={"ID":"2f940874-3c9d-4de3-985d-5d3c69b0edd3","Type":"ContainerDied","Data":"4b4499114c447412b06dd2081f5c6ce55e0795bb55f022f9d305b6f629db1afb"}
Oct 30 21:04:02.986456 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:04:02.979142    4521 generic.go:356] "Generic (PLEG): container finished" podID="fbdd2291-639e-4c1e-986d-425c4ab63dbc" containerID="026db3fbe3f5478b701e10343c4817d9fe47d7df58e5baf741b24758eb686213" exitCode=0
Oct 30 21:04:02.986456 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:04:02.979344    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-lp6sk" event={"ID":"fbdd2291-639e-4c1e-986d-425c4ab63dbc","Type":"ContainerDied","Data":"026db3fbe3f5478b701e10343c4817d9fe47d7df58e5baf741b24758eb686213"}
Oct 30 21:04:03.386351 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:04:03.382915    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-lp6sk"
Oct 30 21:04:03.505099 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:04:03.505062    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbdd2291-639e-4c1e-986d-425c4ab63dbc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbdd2291-639e-4c1e-986d-425c4ab63dbc-istio-data\") pod \"fbdd2291-639e-4c1e-986d-425c4ab63dbc\" (UID: \"fbdd2291-639e-4c1e-986d-425c4ab63dbc\") "
Oct 30 21:04:03.505283 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:04:03.505148    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fbdd2291-639e-4c1e-986d-425c4ab63dbc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbdd2291-639e-4c1e-986d-425c4ab63dbc-istio-podinfo\") pod \"fbdd2291-639e-4c1e-986d-425c4ab63dbc\" (UID: \"fbdd2291-639e-4c1e-986d-425c4ab63dbc\") "
Oct 30 21:04:03.505283 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:04:03.505189    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbdd2291-639e-4c1e-986d-425c4ab63dbc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbdd2291-639e-4c1e-986d-425c4ab63dbc-workload-socket\") pod \"fbdd2291-639e-4c1e-986d-425c4ab63dbc\" (UID: \"fbdd2291-639e-4c1e-986d-425c4ab63dbc\") "
Oct 30 21:04:03.505283 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:04:03.505238    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbdd2291-639e-4c1e-986d-425c4ab63dbc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbdd2291-639e-4c1e-986d-425c4ab63dbc-workload-certs\") pod \"fbdd2291-639e-4c1e-986d-425c4ab63dbc\" (UID: \"fbdd2291-639e-4c1e-986d-425c4ab63dbc\") "
Oct 30 21:04:03.505283 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:04:03.505271    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbdd2291-639e-4c1e-986d-425c4ab63dbc-kube-api-access-w2svb\" (UniqueName: \"kubernetes.io/projected/fbdd2291-639e-4c1e-986d-425c4ab63dbc-kube-api-access-w2svb\") pod \"fbdd2291-639e-4c1e-986d-425c4ab63dbc\" (UID: \"fbdd2291-639e-4c1e-986d-425c4ab63dbc\") "
Oct 30 21:04:03.505607 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:04:03.505359    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbdd2291-639e-4c1e-986d-425c4ab63dbc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbdd2291-639e-4c1e-986d-425c4ab63dbc-istiod-ca-cert\") pod \"fbdd2291-639e-4c1e-986d-425c4ab63dbc\" (UID: \"fbdd2291-639e-4c1e-986d-425c4ab63dbc\") "
Oct 30 21:04:03.505607 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:04:03.505434    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbdd2291-639e-4c1e-986d-425c4ab63dbc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbdd2291-639e-4c1e-986d-425c4ab63dbc-istio-envoy\") pod \"fbdd2291-639e-4c1e-986d-425c4ab63dbc\" (UID: \"fbdd2291-639e-4c1e-986d-425c4ab63dbc\") "
Oct 30 21:04:03.505607 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:04:03.505478    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbdd2291-639e-4c1e-986d-425c4ab63dbc-istio-token\" (UniqueName: \"kubernetes.io/projected/fbdd2291-639e-4c1e-986d-425c4ab63dbc-istio-token\") pod \"fbdd2291-639e-4c1e-986d-425c4ab63dbc\" (UID: \"fbdd2291-639e-4c1e-986d-425c4ab63dbc\") "
Oct 30 21:04:03.505607 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:04:03.505531    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbdd2291-639e-4c1e-986d-425c4ab63dbc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbdd2291-639e-4c1e-986d-425c4ab63dbc-credential-socket\") pod \"fbdd2291-639e-4c1e-986d-425c4ab63dbc\" (UID: \"fbdd2291-639e-4c1e-986d-425c4ab63dbc\") "
Oct 30 21:04:03.508232 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:04:03.508192    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbdd2291-639e-4c1e-986d-425c4ab63dbc-workload-socket" pod "fbdd2291-639e-4c1e-986d-425c4ab63dbc" (UID: "fbdd2291-639e-4c1e-986d-425c4ab63dbc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:04:03.508387 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:04:03.508281    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbdd2291-639e-4c1e-986d-425c4ab63dbc-istiod-ca-cert" pod "fbdd2291-639e-4c1e-986d-425c4ab63dbc" (UID: "fbdd2291-639e-4c1e-986d-425c4ab63dbc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:04:03.508453 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:04:03.508402    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbdd2291-639e-4c1e-986d-425c4ab63dbc-workload-certs" pod "fbdd2291-639e-4c1e-986d-425c4ab63dbc" (UID: "fbdd2291-639e-4c1e-986d-425c4ab63dbc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:04:03.508889 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:04:03.508865    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbdd2291-639e-4c1e-986d-425c4ab63dbc-credential-socket" pod "fbdd2291-639e-4c1e-986d-425c4ab63dbc" (UID: "fbdd2291-639e-4c1e-986d-425c4ab63dbc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:04:03.537365 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:04:03.509351    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbdd2291-639e-4c1e-986d-425c4ab63dbc-istio-data" pod "fbdd2291-639e-4c1e-986d-425c4ab63dbc" (UID: "fbdd2291-639e-4c1e-986d-425c4ab63dbc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:04:03.537365 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:04:03.524526    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fbdd2291-639e-4c1e-986d-425c4ab63dbc-istio-podinfo" pod "fbdd2291-639e-4c1e-986d-425c4ab63dbc" (UID: "fbdd2291-639e-4c1e-986d-425c4ab63dbc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:04:03.538097 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:04:03.538068    4521 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbdd2291-639e-4c1e-986d-425c4ab63dbc-istio-podinfo\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:04:03.538225 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:04:03.538210    4521 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbdd2291-639e-4c1e-986d-425c4ab63dbc-workload-socket\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:04:03.538316 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:04:03.538304    4521 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbdd2291-639e-4c1e-986d-425c4ab63dbc-workload-certs\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:04:03.538446 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:04:03.538432    4521 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbdd2291-639e-4c1e-986d-425c4ab63dbc-istiod-ca-cert\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:04:03.538534 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:04:03.538522    4521 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbdd2291-639e-4c1e-986d-425c4ab63dbc-credential-socket\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:04:03.538699 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:04:03.538683    4521 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbdd2291-639e-4c1e-986d-425c4ab63dbc-istio-data\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:04:03.565423 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:04:03.563314    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbdd2291-639e-4c1e-986d-425c4ab63dbc-kube-api-access-w2svb" pod "fbdd2291-639e-4c1e-986d-425c4ab63dbc" (UID: "fbdd2291-639e-4c1e-986d-425c4ab63dbc"). InnerVolumeSpecName "kube-api-access-w2svb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:04:03.586359 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:04:03.585479    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbdd2291-639e-4c1e-986d-425c4ab63dbc-istio-envoy" pod "fbdd2291-639e-4c1e-986d-425c4ab63dbc" (UID: "fbdd2291-639e-4c1e-986d-425c4ab63dbc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:04:03.593357 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:04:03.587252    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbdd2291-639e-4c1e-986d-425c4ab63dbc-istio-token" pod "fbdd2291-639e-4c1e-986d-425c4ab63dbc" (UID: "fbdd2291-639e-4c1e-986d-425c4ab63dbc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:04:03.597517 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:04:03.597483    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-7mf77"
Oct 30 21:04:03.603231 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:04:03.603208    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-hz8mp"
Oct 30 21:04:03.665370 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:04:03.664379    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b77d06a-b7a8-4b2d-a365-a728d4397d8b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b77d06a-b7a8-4b2d-a365-a728d4397d8b-istio-data\") pod \"5b77d06a-b7a8-4b2d-a365-a728d4397d8b\" (UID: \"5b77d06a-b7a8-4b2d-a365-a728d4397d8b\") "
Oct 30 21:04:03.665370 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:04:03.664433    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b77d06a-b7a8-4b2d-a365-a728d4397d8b-istio-token\" (UniqueName: \"kubernetes.io/projected/5b77d06a-b7a8-4b2d-a365-a728d4397d8b-istio-token\") pod \"5b77d06a-b7a8-4b2d-a365-a728d4397d8b\" (UID: \"5b77d06a-b7a8-4b2d-a365-a728d4397d8b\") "
Oct 30 21:04:03.665370 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:04:03.664493    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f940874-3c9d-4de3-985d-5d3c69b0edd3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f940874-3c9d-4de3-985d-5d3c69b0edd3-credential-socket\") pod \"2f940874-3c9d-4de3-985d-5d3c69b0edd3\" (UID: \"2f940874-3c9d-4de3-985d-5d3c69b0edd3\") "
Oct 30 21:04:03.665370 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:04:03.664551    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f940874-3c9d-4de3-985d-5d3c69b0edd3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f940874-3c9d-4de3-985d-5d3c69b0edd3-istio-envoy\") pod \"2f940874-3c9d-4de3-985d-5d3c69b0edd3\" (UID: \"2f940874-3c9d-4de3-985d-5d3c69b0edd3\") "
Oct 30 21:04:03.665370 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:04:03.664583    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5b77d06a-b7a8-4b2d-a365-a728d4397d8b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b77d06a-b7a8-4b2d-a365-a728d4397d8b-istio-podinfo\") pod \"5b77d06a-b7a8-4b2d-a365-a728d4397d8b\" (UID: \"5b77d06a-b7a8-4b2d-a365-a728d4397d8b\") "
Oct 30 21:04:03.665370 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:04:03.664619    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b77d06a-b7a8-4b2d-a365-a728d4397d8b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b77d06a-b7a8-4b2d-a365-a728d4397d8b-istio-envoy\") pod \"5b77d06a-b7a8-4b2d-a365-a728d4397d8b\" (UID: \"5b77d06a-b7a8-4b2d-a365-a728d4397d8b\") "
Oct 30 21:04:03.665370 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:04:03.664671    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f940874-3c9d-4de3-985d-5d3c69b0edd3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f940874-3c9d-4de3-985d-5d3c69b0edd3-istiod-ca-cert\") pod \"2f940874-3c9d-4de3-985d-5d3c69b0edd3\" (UID: \"2f940874-3c9d-4de3-985d-5d3c69b0edd3\") "
Oct 30 21:04:03.665370 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:04:03.664705    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b77d06a-b7a8-4b2d-a365-a728d4397d8b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b77d06a-b7a8-4b2d-a365-a728d4397d8b-credential-socket\") pod \"5b77d06a-b7a8-4b2d-a365-a728d4397d8b\" (UID: \"5b77d06a-b7a8-4b2d-a365-a728d4397d8b\") "
Oct 30 21:04:03.665370 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:04:03.664738    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f940874-3c9d-4de3-985d-5d3c69b0edd3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f940874-3c9d-4de3-985d-5d3c69b0edd3-istio-data\") pod \"2f940874-3c9d-4de3-985d-5d3c69b0edd3\" (UID: \"2f940874-3c9d-4de3-985d-5d3c69b0edd3\") "
Oct 30 21:04:03.665370 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:04:03.664779    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b77d06a-b7a8-4b2d-a365-a728d4397d8b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b77d06a-b7a8-4b2d-a365-a728d4397d8b-workload-socket\") pod \"5b77d06a-b7a8-4b2d-a365-a728d4397d8b\" (UID: \"5b77d06a-b7a8-4b2d-a365-a728d4397d8b\") "
Oct 30 21:04:03.665370 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:04:03.664807    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f940874-3c9d-4de3-985d-5d3c69b0edd3-istio-token\" (UniqueName: \"kubernetes.io/projected/2f940874-3c9d-4de3-985d-5d3c69b0edd3-istio-token\") pod \"2f940874-3c9d-4de3-985d-5d3c69b0edd3\" (UID: \"2f940874-3c9d-4de3-985d-5d3c69b0edd3\") "
Oct 30 21:04:03.665370 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:04:03.664860    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b77d06a-b7a8-4b2d-a365-a728d4397d8b-kube-api-access-q6bcv\" (UniqueName: \"kubernetes.io/projected/5b77d06a-b7a8-4b2d-a365-a728d4397d8b-kube-api-access-q6bcv\") pod \"5b77d06a-b7a8-4b2d-a365-a728d4397d8b\" (UID: \"5b77d06a-b7a8-4b2d-a365-a728d4397d8b\") "
Oct 30 21:04:03.665370 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:04:03.664892    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b77d06a-b7a8-4b2d-a365-a728d4397d8b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b77d06a-b7a8-4b2d-a365-a728d4397d8b-workload-certs\") pod \"5b77d06a-b7a8-4b2d-a365-a728d4397d8b\" (UID: \"5b77d06a-b7a8-4b2d-a365-a728d4397d8b\") "
Oct 30 21:04:03.665370 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:04:03.664933    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b77d06a-b7a8-4b2d-a365-a728d4397d8b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b77d06a-b7a8-4b2d-a365-a728d4397d8b-istiod-ca-cert\") pod \"5b77d06a-b7a8-4b2d-a365-a728d4397d8b\" (UID: \"5b77d06a-b7a8-4b2d-a365-a728d4397d8b\") "
Oct 30 21:04:03.665370 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:04:03.664984    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f940874-3c9d-4de3-985d-5d3c69b0edd3-kube-api-access-px4hw\" (UniqueName: \"kubernetes.io/projected/2f940874-3c9d-4de3-985d-5d3c69b0edd3-kube-api-access-px4hw\") pod \"2f940874-3c9d-4de3-985d-5d3c69b0edd3\" (UID: \"2f940874-3c9d-4de3-985d-5d3c69b0edd3\") "
Oct 30 21:04:03.665370 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:04:03.665022    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f940874-3c9d-4de3-985d-5d3c69b0edd3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f940874-3c9d-4de3-985d-5d3c69b0edd3-workload-certs\") pod \"2f940874-3c9d-4de3-985d-5d3c69b0edd3\" (UID: \"2f940874-3c9d-4de3-985d-5d3c69b0edd3\") "
Oct 30 21:04:03.665370 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:04:03.665054    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2f940874-3c9d-4de3-985d-5d3c69b0edd3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f940874-3c9d-4de3-985d-5d3c69b0edd3-istio-podinfo\") pod \"2f940874-3c9d-4de3-985d-5d3c69b0edd3\" (UID: \"2f940874-3c9d-4de3-985d-5d3c69b0edd3\") "
Oct 30 21:04:03.665370 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:04:03.665113    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f940874-3c9d-4de3-985d-5d3c69b0edd3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f940874-3c9d-4de3-985d-5d3c69b0edd3-workload-socket\") pod \"2f940874-3c9d-4de3-985d-5d3c69b0edd3\" (UID: \"2f940874-3c9d-4de3-985d-5d3c69b0edd3\") "
Oct 30 21:04:03.666511 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:04:03.665460    4521 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2svb\" (UniqueName: \"kubernetes.io/projected/fbdd2291-639e-4c1e-986d-425c4ab63dbc-kube-api-access-w2svb\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:04:03.666511 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:04:03.665480    4521 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbdd2291-639e-4c1e-986d-425c4ab63dbc-istio-envoy\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:04:03.666511 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:04:03.665493    4521 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbdd2291-639e-4c1e-986d-425c4ab63dbc-istio-token\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:04:03.690351 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:04:03.686816    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b77d06a-b7a8-4b2d-a365-a728d4397d8b-istio-data" pod "5b77d06a-b7a8-4b2d-a365-a728d4397d8b" (UID: "5b77d06a-b7a8-4b2d-a365-a728d4397d8b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:04:03.690351 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:04:03.688706    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b77d06a-b7a8-4b2d-a365-a728d4397d8b-istiod-ca-cert" pod "5b77d06a-b7a8-4b2d-a365-a728d4397d8b" (UID: "5b77d06a-b7a8-4b2d-a365-a728d4397d8b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:04:03.691120 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:04:03.690829    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f940874-3c9d-4de3-985d-5d3c69b0edd3-credential-socket" pod "2f940874-3c9d-4de3-985d-5d3c69b0edd3" (UID: "2f940874-3c9d-4de3-985d-5d3c69b0edd3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:04:03.691258 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:04:03.691137    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b77d06a-b7a8-4b2d-a365-a728d4397d8b-workload-certs" pod "5b77d06a-b7a8-4b2d-a365-a728d4397d8b" (UID: "5b77d06a-b7a8-4b2d-a365-a728d4397d8b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:04:03.691411 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:04:03.691375    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f940874-3c9d-4de3-985d-5d3c69b0edd3-workload-certs" pod "2f940874-3c9d-4de3-985d-5d3c69b0edd3" (UID: "2f940874-3c9d-4de3-985d-5d3c69b0edd3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:04:03.696232 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:04:03.691683    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f940874-3c9d-4de3-985d-5d3c69b0edd3-istiod-ca-cert" pod "2f940874-3c9d-4de3-985d-5d3c69b0edd3" (UID: "2f940874-3c9d-4de3-985d-5d3c69b0edd3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:04:03.696232 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:04:03.692910    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b77d06a-b7a8-4b2d-a365-a728d4397d8b-credential-socket" pod "5b77d06a-b7a8-4b2d-a365-a728d4397d8b" (UID: "5b77d06a-b7a8-4b2d-a365-a728d4397d8b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:04:03.696232 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:04:03.692928    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b77d06a-b7a8-4b2d-a365-a728d4397d8b-istio-token" pod "5b77d06a-b7a8-4b2d-a365-a728d4397d8b" (UID: "5b77d06a-b7a8-4b2d-a365-a728d4397d8b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:04:03.696232 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:04:03.693118    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f940874-3c9d-4de3-985d-5d3c69b0edd3-istio-data" pod "2f940874-3c9d-4de3-985d-5d3c69b0edd3" (UID: "2f940874-3c9d-4de3-985d-5d3c69b0edd3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:04:03.696232 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:04:03.694493    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f940874-3c9d-4de3-985d-5d3c69b0edd3-workload-socket" pod "2f940874-3c9d-4de3-985d-5d3c69b0edd3" (UID: "2f940874-3c9d-4de3-985d-5d3c69b0edd3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:04:03.696232 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:04:03.694578    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b77d06a-b7a8-4b2d-a365-a728d4397d8b-workload-socket" pod "5b77d06a-b7a8-4b2d-a365-a728d4397d8b" (UID: "5b77d06a-b7a8-4b2d-a365-a728d4397d8b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:04:03.696232 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:04:03.695451    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2f940874-3c9d-4de3-985d-5d3c69b0edd3-istio-podinfo" pod "2f940874-3c9d-4de3-985d-5d3c69b0edd3" (UID: "2f940874-3c9d-4de3-985d-5d3c69b0edd3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:04:03.696232 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:04:03.695695    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b77d06a-b7a8-4b2d-a365-a728d4397d8b-istio-envoy" pod "5b77d06a-b7a8-4b2d-a365-a728d4397d8b" (UID: "5b77d06a-b7a8-4b2d-a365-a728d4397d8b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:04:03.696232 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:04:03.695965    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f940874-3c9d-4de3-985d-5d3c69b0edd3-istio-token" pod "2f940874-3c9d-4de3-985d-5d3c69b0edd3" (UID: "2f940874-3c9d-4de3-985d-5d3c69b0edd3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:04:03.698364 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:04:03.698114    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f940874-3c9d-4de3-985d-5d3c69b0edd3-istio-envoy" pod "2f940874-3c9d-4de3-985d-5d3c69b0edd3" (UID: "2f940874-3c9d-4de3-985d-5d3c69b0edd3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:04:03.698364 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:04:03.698200    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b77d06a-b7a8-4b2d-a365-a728d4397d8b-kube-api-access-q6bcv" pod "5b77d06a-b7a8-4b2d-a365-a728d4397d8b" (UID: "5b77d06a-b7a8-4b2d-a365-a728d4397d8b"). InnerVolumeSpecName "kube-api-access-q6bcv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:04:03.698717 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:04:03.698692    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f940874-3c9d-4de3-985d-5d3c69b0edd3-kube-api-access-px4hw" pod "2f940874-3c9d-4de3-985d-5d3c69b0edd3" (UID: "2f940874-3c9d-4de3-985d-5d3c69b0edd3"). InnerVolumeSpecName "kube-api-access-px4hw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:04:03.699773 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:04:03.699747    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5b77d06a-b7a8-4b2d-a365-a728d4397d8b-istio-podinfo" pod "5b77d06a-b7a8-4b2d-a365-a728d4397d8b" (UID: "5b77d06a-b7a8-4b2d-a365-a728d4397d8b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:04:03.769348 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:04:03.768879    4521 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f940874-3c9d-4de3-985d-5d3c69b0edd3-workload-certs\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:04:03.769348 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:04:03.768923    4521 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f940874-3c9d-4de3-985d-5d3c69b0edd3-istio-podinfo\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:04:03.769348 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:04:03.768939    4521 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f940874-3c9d-4de3-985d-5d3c69b0edd3-workload-socket\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:04:03.769348 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:04:03.768953    4521 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b77d06a-b7a8-4b2d-a365-a728d4397d8b-istio-data\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:04:03.769348 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:04:03.768967    4521 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b77d06a-b7a8-4b2d-a365-a728d4397d8b-istio-token\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:04:03.769348 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:04:03.768980    4521 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f940874-3c9d-4de3-985d-5d3c69b0edd3-credential-socket\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:04:03.769348 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:04:03.768992    4521 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f940874-3c9d-4de3-985d-5d3c69b0edd3-istio-envoy\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:04:03.769348 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:04:03.769005    4521 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b77d06a-b7a8-4b2d-a365-a728d4397d8b-istio-podinfo\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:04:03.769348 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:04:03.769019    4521 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b77d06a-b7a8-4b2d-a365-a728d4397d8b-istio-envoy\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:04:03.769348 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:04:03.769032    4521 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f940874-3c9d-4de3-985d-5d3c69b0edd3-istiod-ca-cert\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:04:03.769348 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:04:03.769046    4521 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b77d06a-b7a8-4b2d-a365-a728d4397d8b-credential-socket\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:04:03.769348 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:04:03.769061    4521 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f940874-3c9d-4de3-985d-5d3c69b0edd3-istio-data\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:04:03.769348 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:04:03.769076    4521 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b77d06a-b7a8-4b2d-a365-a728d4397d8b-workload-socket\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:04:03.769348 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:04:03.769089    4521 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f940874-3c9d-4de3-985d-5d3c69b0edd3-istio-token\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:04:03.769348 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:04:03.769103    4521 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6bcv\" (UniqueName: \"kubernetes.io/projected/5b77d06a-b7a8-4b2d-a365-a728d4397d8b-kube-api-access-q6bcv\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:04:03.769348 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:04:03.769119    4521 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b77d06a-b7a8-4b2d-a365-a728d4397d8b-workload-certs\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:04:03.769348 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:04:03.769134    4521 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b77d06a-b7a8-4b2d-a365-a728d4397d8b-istiod-ca-cert\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:04:03.769348 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:04:03.769148    4521 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-px4hw\" (UniqueName: \"kubernetes.io/projected/2f940874-3c9d-4de3-985d-5d3c69b0edd3-kube-api-access-px4hw\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:04:03.989877 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:04:03.989842    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-lp6sk"
Oct 30 21:04:03.989877 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:04:03.989866    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-lp6sk" event={"ID":"fbdd2291-639e-4c1e-986d-425c4ab63dbc","Type":"ContainerDied","Data":"8fc7a41837386490c3daa999639e67973d6e6d173667bd3fc497679dd56a615a"}
Oct 30 21:04:03.990370 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:04:03.989941    4521 scope.go:122] "RemoveContainer" containerID="026db3fbe3f5478b701e10343c4817d9fe47d7df58e5baf741b24758eb686213"
Oct 30 21:04:03.991838 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:04:03.991807    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-hz8mp" event={"ID":"5b77d06a-b7a8-4b2d-a365-a728d4397d8b","Type":"ContainerDied","Data":"620ee961a449611a0340f50fcdefce412cd455cf9730a98fbfb899d909c3f16d"}
Oct 30 21:04:03.991982 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:04:03.991909    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-hz8mp"
Oct 30 21:04:04.001995 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:04:04.001085    4521 scope.go:122] "RemoveContainer" containerID="1fc06542506386cd87be1aa2f62f7c628d987268e35a2f5c5f557c0249431354"
Oct 30 21:04:04.002850 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:04:04.002828    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-7mf77" event={"ID":"2f940874-3c9d-4de3-985d-5d3c69b0edd3","Type":"ContainerDied","Data":"359e894924043d1c36d2b7abea7fa1b6405cf671e64827950e4aa7fae91ba3a6"}
Oct 30 21:04:04.002957 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:04:04.002852    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-7mf77"
Oct 30 21:04:04.060360 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:04:04.060314    4521 scope.go:122] "RemoveContainer" containerID="4b4499114c447412b06dd2081f5c6ce55e0795bb55f022f9d305b6f629db1afb"
Oct 30 21:04:04.067908 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:04:04.067871    4521 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-lp6sk"]
Oct 30 21:04:04.076111 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:04:04.075822    4521 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-lp6sk"]
Oct 30 21:04:04.090925 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:04:04.090642    4521 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-hz8mp"]
Oct 30 21:04:04.095705 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:04:04.095677    4521 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-hz8mp"]
Oct 30 21:04:04.104509 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:04:04.104484    4521 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-7mf77"]
Oct 30 21:04:04.115967 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:04:04.115939    4521 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-c4b379-openshift-default-57fdd6d787-7mf77"]
Oct 30 21:04:04.790970 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:04:04.790930    4521 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f940874-3c9d-4de3-985d-5d3c69b0edd3" path="/var/lib/kubelet/pods/2f940874-3c9d-4de3-985d-5d3c69b0edd3/volumes"
Oct 30 21:04:04.791409 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:04:04.791392    4521 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b77d06a-b7a8-4b2d-a365-a728d4397d8b" path="/var/lib/kubelet/pods/5b77d06a-b7a8-4b2d-a365-a728d4397d8b/volumes"
Oct 30 21:04:04.791790 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:04:04.791772    4521 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbdd2291-639e-4c1e-986d-425c4ab63dbc" path="/var/lib/kubelet/pods/fbdd2291-639e-4c1e-986d-425c4ab63dbc/volumes"
Oct 30 21:04:16.428872 ip-10-0-131-125 kubenswrapper[4521]: E1030 21:04:16.428825    4521 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lkn2k" containerName="extractor"
Oct 30 21:04:25.027885 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:04:25.027833    4521 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52c565cc-f2ppx"]
Oct 30 21:04:25.034019 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:04:25.033971    4521 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52c565cc-f2ppx"]
Oct 30 21:04:26.790983 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:04:26.790947    4521 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06e939c1-2403-439d-b599-ab8ff61a042d" path="/var/lib/kubelet/pods/06e939c1-2403-439d-b599-ab8ff61a042d/volumes"
Oct 30 21:04:34.025788 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:04:34.025740    4521 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf7689f5-sjgsr"]
Oct 30 21:04:34.029367 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:04:34.029337    4521 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf7689f5-sjgsr"]
Oct 30 21:04:34.790922 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:04:34.790881    4521 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82e161c9-b3d1-4448-9cd9-403a0b6ffa3a" path="/var/lib/kubelet/pods/82e161c9-b3d1-4448-9cd9-403a0b6ffa3a/volumes"
Oct 30 21:04:36.031693 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:04:36.031659    4521 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11299c87-dlggc"]
Oct 30 21:04:36.035155 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:04:36.035127    4521 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11299c87-dlggc"]
Oct 30 21:04:36.791144 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:04:36.791111    4521 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eecab7a4-8911-4b5f-b6fa-97727ae3e1cd" path="/var/lib/kubelet/pods/eecab7a4-8911-4b5f-b6fa-97727ae3e1cd/volumes"
Oct 30 21:04:37.026248 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:04:37.026217    4521 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d92a4e2c-mvb6w"]
Oct 30 21:04:37.036466 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:04:37.036428    4521 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d92a4e2c-mvb6w"]
Oct 30 21:04:38.790632 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:04:38.790599    4521 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="680a80b6-b2fd-4aa5-9128-23193a693221" path="/var/lib/kubelet/pods/680a80b6-b2fd-4aa5-9128-23193a693221/volumes"
Oct 30 21:04:45.030523 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:04:45.030482    4521 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe8d59bc-chscd"]
Oct 30 21:04:45.033680 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:04:45.033657    4521 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe8d59bc-chscd"]
Oct 30 21:04:46.429273 ip-10-0-131-125 kubenswrapper[4521]: E1030 21:04:46.429231    4521 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lkn2k" containerName="extractor"
Oct 30 21:04:46.791343 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:04:46.791296    4521 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="737f9761-4c67-41ca-9311-904bbfc50df4" path="/var/lib/kubelet/pods/737f9761-4c67-41ca-9311-904bbfc50df4/volumes"
Oct 30 21:04:48.026368 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:04:48.026314    4521 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd72df56-n6dtl"]
Oct 30 21:04:48.032208 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:04:48.032173    4521 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd72df56-n6dtl"]
Oct 30 21:04:48.791287 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:04:48.791248    4521 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="769224a3-a1ff-458c-b43b-c802b6311ba4" path="/var/lib/kubelet/pods/769224a3-a1ff-458c-b43b-c802b6311ba4/volumes"
Oct 30 21:04:51.025337 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:04:51.025294    4521 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5a591e6-d2vtx"]
Oct 30 21:04:51.034109 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:04:51.034070    4521 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5a591e6-d2vtx"]
Oct 30 21:04:52.790810 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:04:52.790776    4521 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1016070-22d2-4278-8fe0-782cb3608dcf" path="/var/lib/kubelet/pods/c1016070-22d2-4278-8fe0-782cb3608dcf/volumes"
Oct 30 21:04:53.024937 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:04:53.024906    4521 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3c3bec5-mv7m5"]
Oct 30 21:04:53.027792 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:04:53.027770    4521 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3c3bec5-mv7m5"]
Oct 30 21:04:53.322837 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:04:53.322807    4521 scope.go:122] "RemoveContainer" containerID="ecc44c3983a6afc4815071bbc2e79af31a0e055659cee676c6271a9969018ae1"
Oct 30 21:04:54.790992 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:04:54.790956    4521 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06a6a3a4-f7a3-4235-a80b-8e2163968e35" path="/var/lib/kubelet/pods/06a6a3a4-f7a3-4235-a80b-8e2163968e35/volumes"
Oct 30 21:05:02.025545 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:05:02.025501    4521 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e488030-ngvh7"]
Oct 30 21:05:02.028578 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:05:02.028549    4521 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e488030-ngvh7"]
Oct 30 21:05:02.792025 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:05:02.791983    4521 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3665eb7-48ee-4eef-a8c9-06e5ebfec690" path="/var/lib/kubelet/pods/d3665eb7-48ee-4eef-a8c9-06e5ebfec690/volumes"
Oct 30 21:05:09.321927 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:05:09.321884    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4kxb"]
Oct 30 21:05:16.428770 ip-10-0-131-125 kubenswrapper[4521]: E1030 21:05:16.428732    4521 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lkn2k" containerName="extractor"
Oct 30 21:05:21.027207 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:05:21.027171    4521 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2762bcd6-4j45v"]
Oct 30 21:05:21.031776 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:05:21.031747    4521 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2762bcd6-4j45v"]
Oct 30 21:05:22.796825 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:05:22.796791    4521 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4149346b-deed-426d-9b80-e47822d8b731" path="/var/lib/kubelet/pods/4149346b-deed-426d-9b80-e47822d8b731/volumes"
Oct 30 21:05:25.012016 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:05:25.011974    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4kxb"]
Oct 30 21:05:29.786161 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:05:29.786126    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4kxb"]
Oct 30 21:05:45.989381 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:05:45.989343    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4kxb"]
Oct 30 21:05:46.431172 ip-10-0-131-125 kubenswrapper[4521]: E1030 21:05:46.431064    4521 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lkn2k" containerName="extractor"
Oct 30 21:05:53.372002 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:05:53.371964    4521 scope.go:122] "RemoveContainer" containerID="d5a2583ce54c0167f86336c241e0152bfbe1b1ad07d8eae2e2b4aee8bc397c66"
Oct 30 21:05:53.381575 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:05:53.381549    4521 scope.go:122] "RemoveContainer" containerID="daa972aea978ae6626914a284ac2ec68cf2930b5f5590660d25a7420cd5df4d5"
Oct 30 21:05:53.393089 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:05:53.393065    4521 scope.go:122] "RemoveContainer" containerID="3fc4626481d2cecd58e9b2aae3fda4080331b32837a935da4cc89c8144b36353"
Oct 30 21:05:53.401744 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:05:53.401724    4521 scope.go:122] "RemoveContainer" containerID="0418f0b50bc6be455544a63d9b0f7a5b46a38b2cfb0505813ba8049f9afc2596"
Oct 30 21:05:53.410021 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:05:53.409999    4521 scope.go:122] "RemoveContainer" containerID="e20eda67237e3bb82f232e8d05704f547e15efe2dc1cf956bab94bc206775893"
Oct 30 21:05:53.418381 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:05:53.418361    4521 scope.go:122] "RemoveContainer" containerID="d2a53a9d6acd94400a50aac4040bf09b7743f31404303a3a558c37d721f1d052"
Oct 30 21:05:53.426622 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:05:53.426603    4521 scope.go:122] "RemoveContainer" containerID="59ce9780cf45903b387544fef5b2a1fe8b0b84f8f3b7e3ceb033000e23a67e83"
Oct 30 21:05:53.435129 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:05:53.435108    4521 scope.go:122] "RemoveContainer" containerID="c57dbe64ca675865d5af4b195ff48207773f7c234f161c2cd3954776a7ad6a88"
Oct 30 21:05:57.029313 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:05:57.029272    4521 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59c65aca-t465g"]
Oct 30 21:05:57.032844 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:05:57.032815    4521 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59c65aca-t465g"]
Oct 30 21:05:58.791464 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:05:58.791432    4521 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa6b7076-9843-483f-94ce-e61e47536580" path="/var/lib/kubelet/pods/fa6b7076-9843-483f-94ce-e61e47536580/volumes"
Oct 30 21:06:00.892365 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:06:00.892306    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4kxb"]
Oct 30 21:06:04.163964 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:06:04.163919    4521 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-sq68x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:06:04.164496 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:06:04.163995    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-sq68x" podUID="16e33702-94a7-4c37-b430-1ac4f002832b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:06:09.035719 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:06:09.035681    4521 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-847c3a60-bzqgs"]
Oct 30 21:06:09.038495 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:06:09.038459    4521 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-847c3a60-bzqgs"]
Oct 30 21:06:10.790856 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:06:10.790821    4521 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="655b039a-00a8-40fd-baaa-e58a80149a06" path="/var/lib/kubelet/pods/655b039a-00a8-40fd-baaa-e58a80149a06/volumes"
Oct 30 21:06:16.428889 ip-10-0-131-125 kubenswrapper[4521]: E1030 21:06:16.428846    4521 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lkn2k" containerName="extractor"
Oct 30 21:06:17.029165 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:06:17.029121    4521 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9bbfd6c-flhst"]
Oct 30 21:06:17.032934 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:06:17.032906    4521 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9bbfd6c-flhst"]
Oct 30 21:06:18.791276 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:06:18.791241    4521 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84b77cef-b726-42b7-9917-8e3cd108bcef" path="/var/lib/kubelet/pods/84b77cef-b726-42b7-9917-8e3cd108bcef/volumes"
Oct 30 21:06:25.411094 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:06:25.411053    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4kxb"]
Oct 30 21:06:35.530228 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:06:35.530127    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4kxb"]
Oct 30 21:06:37.031746 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:06:37.031710    4521 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f768d99c-rp29v"]
Oct 30 21:06:37.034484 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:06:37.034458    4521 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f768d99c-rp29v"]
Oct 30 21:06:38.797522 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:06:38.797487    4521 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dfcd9487-cee0-4474-853f-29b908b88366" path="/var/lib/kubelet/pods/dfcd9487-cee0-4474-853f-29b908b88366/volumes"
Oct 30 21:06:40.164342 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:06:40.164292    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4kxb"]
Oct 30 21:06:41.024225 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:06:41.024193    4521 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6880f37-l42pm"]
Oct 30 21:06:41.026891 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:06:41.026869    4521 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6880f37-l42pm"]
Oct 30 21:06:42.027725 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:06:42.024780    4521 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bad53a71-964rp"]
Oct 30 21:06:42.034164 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:06:42.034130    4521 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bad53a71-964rp"]
Oct 30 21:06:42.791004 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:06:42.790968    4521 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="121ec831-c071-4e2e-a05c-24f4aa974505" path="/var/lib/kubelet/pods/121ec831-c071-4e2e-a05c-24f4aa974505/volumes"
Oct 30 21:06:42.791419 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:06:42.791404    4521 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19bae71e-c48b-4b51-9e5c-ea7ce7735057" path="/var/lib/kubelet/pods/19bae71e-c48b-4b51-9e5c-ea7ce7735057/volumes"
Oct 30 21:06:46.023079 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:06:46.023024    4521 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19cf1ad9-gkr8q"]
Oct 30 21:06:46.026343 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:06:46.026303    4521 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19cf1ad9-gkr8q"]
Oct 30 21:06:46.429302 ip-10-0-131-125 kubenswrapper[4521]: E1030 21:06:46.429224    4521 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lkn2k" containerName="extractor"
Oct 30 21:06:46.791572 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:06:46.791536    4521 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc6e70c9-ff92-4d6b-bf03-28441c0750c3" path="/var/lib/kubelet/pods/dc6e70c9-ff92-4d6b-bf03-28441c0750c3/volumes"
Oct 30 21:06:53.488663 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:06:53.488629    4521 scope.go:122] "RemoveContainer" containerID="ec30aab6c0ae6126f6e198b644f2bcc5738d6823efd7ef36cca8326416a01eb2"
Oct 30 21:06:53.497822 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:06:53.497800    4521 scope.go:122] "RemoveContainer" containerID="0998c212c96ef56ae77040422eb0f141d6c4a819f3488164f3639521a2317857"
Oct 30 21:06:53.505861 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:06:53.505844    4521 scope.go:122] "RemoveContainer" containerID="21a3de3b202c4f10c9229e21a63d3e7b1b879d9996e96d4490513f2692dea48a"
Oct 30 21:06:53.513836 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:06:53.513816    4521 scope.go:122] "RemoveContainer" containerID="5ecfb1b5956a52eaf146dec11f6db14e3bba96d43a1549bd12ecfbce3da5220b"
Oct 30 21:06:56.276496 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:06:56.276450    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4kxb"]
Oct 30 21:06:59.904703 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:06:59.904665    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4kxb"]
Oct 30 21:07:06.046839 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:07:06.046803    4521 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-261c359b-nxwd6"]
Oct 30 21:07:06.054880 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:07:06.054850    4521 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-261c359b-nxwd6"]
Oct 30 21:07:06.790877 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:07:06.790841    4521 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="233fc24c-d8c6-4aab-9d22-a68b05a9b378" path="/var/lib/kubelet/pods/233fc24c-d8c6-4aab-9d22-a68b05a9b378/volumes"
Oct 30 21:07:11.025135 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:07:11.025098    4521 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63ecfd7f-vjzz8"]
Oct 30 21:07:11.027873 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:07:11.027844    4521 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63ecfd7f-vjzz8"]
Oct 30 21:07:12.791102 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:07:12.791070    4521 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc97df13-49df-4ee8-bcde-b775f4f594cd" path="/var/lib/kubelet/pods/fc97df13-49df-4ee8-bcde-b775f4f594cd/volumes"
Oct 30 21:07:13.024622 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:07:13.024588    4521 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b875f90-b66l4"]
Oct 30 21:07:13.027149 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:07:13.027123    4521 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b875f90-b66l4"]
Oct 30 21:07:14.026268 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:07:14.026227    4521 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1948999-jnnvm"]
Oct 30 21:07:14.028539 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:07:14.028513    4521 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1948999-jnnvm"]
Oct 30 21:07:14.597802 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:07:14.597765    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4kxb"]
Oct 30 21:07:14.793025 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:07:14.792987    4521 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c6d7c2b-3302-4066-b0e5-842f5d549b4a" path="/var/lib/kubelet/pods/0c6d7c2b-3302-4066-b0e5-842f5d549b4a/volumes"
Oct 30 21:07:14.793613 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:07:14.793595    4521 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1c83c6d-f821-4a73-8747-b8b45e5f346f" path="/var/lib/kubelet/pods/c1c83c6d-f821-4a73-8747-b8b45e5f346f/volumes"
Oct 30 21:07:16.430123 ip-10-0-131-125 kubenswrapper[4521]: E1030 21:07:16.430050    4521 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lkn2k" containerName="extractor"
Oct 30 21:07:19.194351 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:07:19.191447    4521 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-sq68x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:07:19.194351 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:07:19.191520    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-sq68x" podUID="16e33702-94a7-4c37-b430-1ac4f002832b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:07:19.852980 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:07:19.852943    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4kxb"]
Oct 30 21:07:20.024559 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:07:20.024519    4521 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90fc8705-ckxls"]
Oct 30 21:07:20.027584 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:07:20.027550    4521 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba2d654d-mkl4j"]
Oct 30 21:07:20.030107 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:07:20.030082    4521 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90fc8705-ckxls"]
Oct 30 21:07:20.032158 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:07:20.032130    4521 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba2d654d-mkl4j"]
Oct 30 21:07:20.790535 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:07:20.790497    4521 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2894673f-f88e-4d29-840c-982cd4e217bc" path="/var/lib/kubelet/pods/2894673f-f88e-4d29-840c-982cd4e217bc/volumes"
Oct 30 21:07:20.790924 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:07:20.790903    4521 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="867c8cde-f176-451a-b65b-939652b2f312" path="/var/lib/kubelet/pods/867c8cde-f176-451a-b65b-939652b2f312/volumes"
Oct 30 21:07:30.023257 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:07:30.023215    4521 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-214c7679-bnjxj"]
Oct 30 21:07:30.025665 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:07:30.025641    4521 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-214c7679-bnjxj"]
Oct 30 21:07:30.790507 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:07:30.790471    4521 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7bd27e7e-be97-4796-84dd-72f42b5bf42e" path="/var/lib/kubelet/pods/7bd27e7e-be97-4796-84dd-72f42b5bf42e/volumes"
Oct 30 21:07:31.025284 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:07:31.025248    4521 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6f6a826-ggh4m"]
Oct 30 21:07:31.027727 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:07:31.027705    4521 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6f6a826-ggh4m"]
Oct 30 21:07:32.791402 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:07:32.791368    4521 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b9a4c83-9045-460b-bec6-8c2a6c3615a3" path="/var/lib/kubelet/pods/9b9a4c83-9045-460b-bec6-8c2a6c3615a3/volumes"
Oct 30 21:07:33.201423 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:07:33.201389    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4kxb"]
Oct 30 21:07:37.067346 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:07:37.066652    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4kxb"]
Oct 30 21:07:46.428519 ip-10-0-131-125 kubenswrapper[4521]: E1030 21:07:46.428482    4521 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lkn2k" containerName="extractor"
Oct 30 21:07:50.993224 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:07:50.993185    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4kxb"]
Oct 30 21:07:51.025317 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:07:51.025276    4521 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-408918eb-f26js"]
Oct 30 21:07:51.027574 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:07:51.027553    4521 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-408918eb-f26js"]
Oct 30 21:07:52.791354 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:07:52.791290    4521 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d10d07e-a60e-4c6a-b067-66234939fa57" path="/var/lib/kubelet/pods/7d10d07e-a60e-4c6a-b067-66234939fa57/volumes"
Oct 30 21:07:53.022940 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:07:53.022907    4521 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-72ee85f3-2hq26"]
Oct 30 21:07:53.028776 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:07:53.027606    4521 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-617df4d2-t4ck6"]
Oct 30 21:07:53.031260 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:07:53.031233    4521 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-72ee85f3-2hq26"]
Oct 30 21:07:53.033457 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:07:53.033433    4521 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-617df4d2-t4ck6"]
Oct 30 21:07:53.549275 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:07:53.549238    4521 scope.go:122] "RemoveContainer" containerID="9fcfadf36e0ec120d8224f77357065db91c3d84d6fb9ea5c65d608fdfef742b6"
Oct 30 21:07:53.563606 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:07:53.563571    4521 scope.go:122] "RemoveContainer" containerID="7e134bd3e9ae8eb8c9af61c92d85f54bc8f6e034ab4f9fcbca50396793babf02"
Oct 30 21:07:53.573737 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:07:53.573714    4521 scope.go:122] "RemoveContainer" containerID="8dbb906f9eeaf3f6deafd55bfc955340c0295d9dcf47ba601fba1c476c89fec6"
Oct 30 21:07:53.582108 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:07:53.582086    4521 scope.go:122] "RemoveContainer" containerID="3e2e97d63e66c394818e2e06282f16a792a8be5acfb6030ebb53b47e8cd80d7d"
Oct 30 21:07:53.590612 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:07:53.590592    4521 scope.go:122] "RemoveContainer" containerID="68fe5fde7b1039f5b4d181fbca9c2514dcf4dc2d91a80f36f717a3125bc91654"
Oct 30 21:07:53.599475 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:07:53.599456    4521 scope.go:122] "RemoveContainer" containerID="154f07e6bcb5d2c20bd1082b0e7eae54ff53298198e064d549d192043eca663b"
Oct 30 21:07:53.607930 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:07:53.607909    4521 scope.go:122] "RemoveContainer" containerID="d48ab49fd65886e8985a68162f760a23b8f74b1c53bd5d1137d534f014c66d85"
Oct 30 21:07:53.616526 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:07:53.616503    4521 scope.go:122] "RemoveContainer" containerID="b3bea3711ed721cb76ccfc76f061ab00a36a59930056e05e1f24814db86d448e"
Oct 30 21:07:53.624955 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:07:53.624931    4521 scope.go:122] "RemoveContainer" containerID="9d7eaffb7e30a7ea1b321a3b58afe2ef35fcb06713518b8492d9d1336fe1ced9"
Oct 30 21:07:54.790894 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:07:54.790861    4521 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e857e76-3d48-4e5b-a8b5-830cfac13f90" path="/var/lib/kubelet/pods/1e857e76-3d48-4e5b-a8b5-830cfac13f90/volumes"
Oct 30 21:07:54.791358 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:07:54.791343    4521 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75d8f639-8cee-4be3-af28-eb7886627983" path="/var/lib/kubelet/pods/75d8f639-8cee-4be3-af28-eb7886627983/volumes"
Oct 30 21:08:01.674139 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:08:01.674105    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4kxb"]
Oct 30 21:08:04.129351 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:08:04.128418    4521 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-54dql container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:08:04.129351 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:08:04.128504    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-54dql" podUID="be342bb0-ebf7-4bb3-a929-5abfc455531e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:08:04.188718 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:08:04.188680    4521 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-sq68x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:08:04.189012 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:08:04.188981    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-sq68x" podUID="16e33702-94a7-4c37-b430-1ac4f002832b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:08:06.230562 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:08:06.220298    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4kxb"]
Oct 30 21:08:16.428970 ip-10-0-131-125 kubenswrapper[4521]: E1030 21:08:16.428924    4521 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lkn2k" containerName="extractor"
Oct 30 21:08:22.825363 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:08:22.825312    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4kxb"]
Oct 30 21:08:26.224832 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:08:26.224791    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4kxb"]
Oct 30 21:08:33.180319 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:08:33.180285    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4kxb"]
Oct 30 21:08:35.659358 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:08:35.658572    4521 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nsppg"]
Oct 30 21:08:35.659358 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:08:35.658825    4521 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nsppg" podUID="47a667bb-4dac-484a-9f1b-01ef17c79710" containerName="discovery" containerID="cri-o://c7c86af13ef1f3f770dbe47e5510e2f31901d0a3ea8c310e29023f3024a816a4" gracePeriod=30
Oct 30 21:08:36.184140 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:08:36.127174    4521 generic.go:356] "Generic (PLEG): container finished" podID="47a667bb-4dac-484a-9f1b-01ef17c79710" containerID="c7c86af13ef1f3f770dbe47e5510e2f31901d0a3ea8c310e29023f3024a816a4" exitCode=0
Oct 30 21:08:36.184140 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:08:36.127433    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nsppg" event={"ID":"47a667bb-4dac-484a-9f1b-01ef17c79710","Type":"ContainerDied","Data":"c7c86af13ef1f3f770dbe47e5510e2f31901d0a3ea8c310e29023f3024a816a4"}
Oct 30 21:08:36.229693 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:08:36.229655    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nsppg"
Oct 30 21:08:36.338430 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:08:36.338306    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/47a667bb-4dac-484a-9f1b-01ef17c79710-cacerts\" (UniqueName: \"kubernetes.io/secret/47a667bb-4dac-484a-9f1b-01ef17c79710-cacerts\") pod \"47a667bb-4dac-484a-9f1b-01ef17c79710\" (UID: \"47a667bb-4dac-484a-9f1b-01ef17c79710\") "
Oct 30 21:08:36.338430 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:08:36.338384    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47a667bb-4dac-484a-9f1b-01ef17c79710-istio-token\" (UniqueName: \"kubernetes.io/projected/47a667bb-4dac-484a-9f1b-01ef17c79710-istio-token\") pod \"47a667bb-4dac-484a-9f1b-01ef17c79710\" (UID: \"47a667bb-4dac-484a-9f1b-01ef17c79710\") "
Oct 30 21:08:36.338675 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:08:36.338446    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47a667bb-4dac-484a-9f1b-01ef17c79710-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/47a667bb-4dac-484a-9f1b-01ef17c79710-local-certs\") pod \"47a667bb-4dac-484a-9f1b-01ef17c79710\" (UID: \"47a667bb-4dac-484a-9f1b-01ef17c79710\") "
Oct 30 21:08:36.338675 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:08:36.338556    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47a667bb-4dac-484a-9f1b-01ef17c79710-kube-api-access-xnvs8\" (UniqueName: \"kubernetes.io/projected/47a667bb-4dac-484a-9f1b-01ef17c79710-kube-api-access-xnvs8\") pod \"47a667bb-4dac-484a-9f1b-01ef17c79710\" (UID: \"47a667bb-4dac-484a-9f1b-01ef17c79710\") "
Oct 30 21:08:36.338675 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:08:36.338596    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/47a667bb-4dac-484a-9f1b-01ef17c79710-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/47a667bb-4dac-484a-9f1b-01ef17c79710-istio-kubeconfig\") pod \"47a667bb-4dac-484a-9f1b-01ef17c79710\" (UID: \"47a667bb-4dac-484a-9f1b-01ef17c79710\") "
Oct 30 21:08:36.338675 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:08:36.338627    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47a667bb-4dac-484a-9f1b-01ef17c79710-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/47a667bb-4dac-484a-9f1b-01ef17c79710-istio-csr-ca-configmap\") pod \"47a667bb-4dac-484a-9f1b-01ef17c79710\" (UID: \"47a667bb-4dac-484a-9f1b-01ef17c79710\") "
Oct 30 21:08:36.338893 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:08:36.338716    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/47a667bb-4dac-484a-9f1b-01ef17c79710-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/47a667bb-4dac-484a-9f1b-01ef17c79710-istio-csr-dns-cert\") pod \"47a667bb-4dac-484a-9f1b-01ef17c79710\" (UID: \"47a667bb-4dac-484a-9f1b-01ef17c79710\") "
Oct 30 21:08:36.339701 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:08:36.339413    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47a667bb-4dac-484a-9f1b-01ef17c79710-istio-csr-ca-configmap" pod "47a667bb-4dac-484a-9f1b-01ef17c79710" (UID: "47a667bb-4dac-484a-9f1b-01ef17c79710"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:08:36.341996 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:08:36.341963    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47a667bb-4dac-484a-9f1b-01ef17c79710-istio-csr-dns-cert" pod "47a667bb-4dac-484a-9f1b-01ef17c79710" (UID: "47a667bb-4dac-484a-9f1b-01ef17c79710"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:08:36.342580 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:08:36.342446    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47a667bb-4dac-484a-9f1b-01ef17c79710-istio-kubeconfig" pod "47a667bb-4dac-484a-9f1b-01ef17c79710" (UID: "47a667bb-4dac-484a-9f1b-01ef17c79710"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:08:36.342580 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:08:36.342540    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47a667bb-4dac-484a-9f1b-01ef17c79710-cacerts" pod "47a667bb-4dac-484a-9f1b-01ef17c79710" (UID: "47a667bb-4dac-484a-9f1b-01ef17c79710"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:08:36.342746 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:08:36.342695    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47a667bb-4dac-484a-9f1b-01ef17c79710-local-certs" pod "47a667bb-4dac-484a-9f1b-01ef17c79710" (UID: "47a667bb-4dac-484a-9f1b-01ef17c79710"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:08:36.342854 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:08:36.342832    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47a667bb-4dac-484a-9f1b-01ef17c79710-istio-token" pod "47a667bb-4dac-484a-9f1b-01ef17c79710" (UID: "47a667bb-4dac-484a-9f1b-01ef17c79710"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:08:36.343606 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:08:36.343585    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47a667bb-4dac-484a-9f1b-01ef17c79710-kube-api-access-xnvs8" pod "47a667bb-4dac-484a-9f1b-01ef17c79710" (UID: "47a667bb-4dac-484a-9f1b-01ef17c79710"). InnerVolumeSpecName "kube-api-access-xnvs8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:08:36.440466 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:08:36.440431    4521 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xnvs8\" (UniqueName: \"kubernetes.io/projected/47a667bb-4dac-484a-9f1b-01ef17c79710-kube-api-access-xnvs8\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:08:36.440466 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:08:36.440464    4521 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/47a667bb-4dac-484a-9f1b-01ef17c79710-istio-kubeconfig\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:08:36.440671 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:08:36.440479    4521 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/47a667bb-4dac-484a-9f1b-01ef17c79710-istio-csr-ca-configmap\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:08:36.440671 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:08:36.440491    4521 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/47a667bb-4dac-484a-9f1b-01ef17c79710-istio-csr-dns-cert\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:08:36.440671 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:08:36.440500    4521 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/47a667bb-4dac-484a-9f1b-01ef17c79710-cacerts\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:08:36.440671 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:08:36.440510    4521 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47a667bb-4dac-484a-9f1b-01ef17c79710-istio-token\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:08:36.440671 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:08:36.440519    4521 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/47a667bb-4dac-484a-9f1b-01ef17c79710-local-certs\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:08:37.135237 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:08:37.135142    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nsppg" event={"ID":"47a667bb-4dac-484a-9f1b-01ef17c79710","Type":"ContainerDied","Data":"43efee53b6c8cc9e2eedf25dbda8715a27c4e49fe2f4096db54fb74e3002d23b"}
Oct 30 21:08:37.135237 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:08:37.135173    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nsppg"
Oct 30 21:08:37.135237 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:08:37.135203    4521 scope.go:122] "RemoveContainer" containerID="c7c86af13ef1f3f770dbe47e5510e2f31901d0a3ea8c310e29023f3024a816a4"
Oct 30 21:08:37.156053 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:08:37.156023    4521 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nsppg"]
Oct 30 21:08:37.160335 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:08:37.160302    4521 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nsppg"]
Oct 30 21:08:38.790545 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:08:38.790511    4521 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47a667bb-4dac-484a-9f1b-01ef17c79710" path="/var/lib/kubelet/pods/47a667bb-4dac-484a-9f1b-01ef17c79710/volumes"
Oct 30 21:08:39.036470 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:08:39.036433    4521 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5fefb0b-nwgfc"]
Oct 30 21:08:39.040489 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:08:39.040456    4521 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5fefb0b-nwgfc"]
Oct 30 21:08:40.717944 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:08:40.717912    4521 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-zbn6s"]
Oct 30 21:08:40.718461 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:08:40.718442    4521 cpu_manager.go:414] "containerMap: removing container" podUID="080a654d-a5bf-4845-a07f-cc8a7e6d0ace" containerName="main"
Oct 30 21:08:40.718537 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:08:40.718463    4521 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="080a654d-a5bf-4845-a07f-cc8a7e6d0ace" containerName="main"
Oct 30 21:08:40.718537 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:08:40.718477    4521 cpu_manager.go:414] "containerMap: removing container" podUID="47a667bb-4dac-484a-9f1b-01ef17c79710" containerName="discovery"
Oct 30 21:08:40.718537 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:08:40.718484    4521 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47a667bb-4dac-484a-9f1b-01ef17c79710" containerName="discovery"
Oct 30 21:08:40.718537 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:08:40.718495    4521 cpu_manager.go:414] "containerMap: removing container" podUID="2f940874-3c9d-4de3-985d-5d3c69b0edd3" containerName="istio-proxy"
Oct 30 21:08:40.718537 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:08:40.718503    4521 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f940874-3c9d-4de3-985d-5d3c69b0edd3" containerName="istio-proxy"
Oct 30 21:08:40.718537 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:08:40.718518    4521 cpu_manager.go:414] "containerMap: removing container" podUID="77f3a44b-d4f2-4506-b386-7a216ee118d9" containerName="payload-pre-processing"
Oct 30 21:08:40.718537 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:08:40.718527    4521 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77f3a44b-d4f2-4506-b386-7a216ee118d9" containerName="payload-pre-processing"
Oct 30 21:08:40.718537 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:08:40.718537    4521 cpu_manager.go:414] "containerMap: removing container" podUID="2aac3e8e-d631-4b9c-bdbb-3b4b1df133a9" containerName="main"
Oct 30 21:08:40.718940 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:08:40.718545    4521 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2aac3e8e-d631-4b9c-bdbb-3b4b1df133a9" containerName="main"
Oct 30 21:08:40.718940 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:08:40.718571    4521 cpu_manager.go:414] "containerMap: removing container" podUID="fbdd2291-639e-4c1e-986d-425c4ab63dbc" containerName="istio-proxy"
Oct 30 21:08:40.718940 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:08:40.718579    4521 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbdd2291-639e-4c1e-986d-425c4ab63dbc" containerName="istio-proxy"
Oct 30 21:08:40.718940 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:08:40.718599    4521 cpu_manager.go:414] "containerMap: removing container" podUID="584d633e-1e52-4eb5-a159-e64d2b9d37e7" containerName="revoke-keys"
Oct 30 21:08:40.718940 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:08:40.718607    4521 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="584d633e-1e52-4eb5-a159-e64d2b9d37e7" containerName="revoke-keys"
Oct 30 21:08:40.718940 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:08:40.718624    4521 cpu_manager.go:414] "containerMap: removing container" podUID="c29ead58-26c1-4720-ac49-99b5e045b85d" containerName="payload-processing"
Oct 30 21:08:40.718940 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:08:40.718631    4521 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c29ead58-26c1-4720-ac49-99b5e045b85d" containerName="payload-processing"
Oct 30 21:08:40.718940 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:08:40.718640    4521 cpu_manager.go:414] "containerMap: removing container" podUID="5b77d06a-b7a8-4b2d-a365-a728d4397d8b" containerName="istio-proxy"
Oct 30 21:08:40.718940 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:08:40.718647    4521 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b77d06a-b7a8-4b2d-a365-a728d4397d8b" containerName="istio-proxy"
Oct 30 21:08:40.718940 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:08:40.718724    4521 memory_manager.go:365] "RemoveStaleState removing state" podUID="77f3a44b-d4f2-4506-b386-7a216ee118d9" containerName="payload-pre-processing"
Oct 30 21:08:40.718940 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:08:40.718736    4521 memory_manager.go:365] "RemoveStaleState removing state" podUID="2aac3e8e-d631-4b9c-bdbb-3b4b1df133a9" containerName="main"
Oct 30 21:08:40.718940 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:08:40.718746    4521 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbdd2291-639e-4c1e-986d-425c4ab63dbc" containerName="istio-proxy"
Oct 30 21:08:40.718940 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:08:40.718755    4521 memory_manager.go:365] "RemoveStaleState removing state" podUID="584d633e-1e52-4eb5-a159-e64d2b9d37e7" containerName="revoke-keys"
Oct 30 21:08:40.718940 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:08:40.718764    4521 memory_manager.go:365] "RemoveStaleState removing state" podUID="c29ead58-26c1-4720-ac49-99b5e045b85d" containerName="payload-processing"
Oct 30 21:08:40.718940 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:08:40.718772    4521 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b77d06a-b7a8-4b2d-a365-a728d4397d8b" containerName="istio-proxy"
Oct 30 21:08:40.718940 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:08:40.718780    4521 memory_manager.go:365] "RemoveStaleState removing state" podUID="080a654d-a5bf-4845-a07f-cc8a7e6d0ace" containerName="main"
Oct 30 21:08:40.718940 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:08:40.718789    4521 memory_manager.go:365] "RemoveStaleState removing state" podUID="47a667bb-4dac-484a-9f1b-01ef17c79710" containerName="discovery"
Oct 30 21:08:40.718940 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:08:40.718796    4521 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f940874-3c9d-4de3-985d-5d3c69b0edd3" containerName="istio-proxy"
Oct 30 21:08:40.721690 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:08:40.721670    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-zbn6s"
Oct 30 21:08:40.729304 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:08:40.729280    4521 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:08:40.729448 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:08:40.729386    4521 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:08:40.729513 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:08:40.729490    4521 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c45pw\""
Oct 30 21:08:40.729795 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:08:40.729776    4521 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:08:40.732515 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:08:40.732497    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-zbn6s"]
Oct 30 21:08:40.779796 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:08:40.779760    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmzbw\" (UniqueName: \"kubernetes.io/projected/9fc825c3-c94c-452c-9cfe-96907a565821-kube-api-access-cmzbw\") pod \"maas-api-revoke-sub-e2fc9c5b-zbn6s\" (UID: \"9fc825c3-c94c-452c-9cfe-96907a565821\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-zbn6s"
Oct 30 21:08:40.779942 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:08:40.779808    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fc825c3-c94c-452c-9cfe-96907a565821-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-zbn6s\" (UID: \"9fc825c3-c94c-452c-9cfe-96907a565821\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-zbn6s"
Oct 30 21:08:40.780004 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:08:40.779951    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fc825c3-c94c-452c-9cfe-96907a565821-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-zbn6s\" (UID: \"9fc825c3-c94c-452c-9cfe-96907a565821\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-zbn6s"
Oct 30 21:08:40.791793 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:08:40.791764    4521 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="584d633e-1e52-4eb5-a159-e64d2b9d37e7" path="/var/lib/kubelet/pods/584d633e-1e52-4eb5-a159-e64d2b9d37e7/volumes"
Oct 30 21:08:40.880803 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:08:40.880769    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fc825c3-c94c-452c-9cfe-96907a565821-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-zbn6s\" (UID: \"9fc825c3-c94c-452c-9cfe-96907a565821\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-zbn6s"
Oct 30 21:08:40.880995 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:08:40.880824    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmzbw\" (UniqueName: \"kubernetes.io/projected/9fc825c3-c94c-452c-9cfe-96907a565821-kube-api-access-cmzbw\") pod \"maas-api-revoke-sub-e2fc9c5b-zbn6s\" (UID: \"9fc825c3-c94c-452c-9cfe-96907a565821\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-zbn6s"
Oct 30 21:08:40.880995 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:08:40.880857    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fc825c3-c94c-452c-9cfe-96907a565821-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-zbn6s\" (UID: \"9fc825c3-c94c-452c-9cfe-96907a565821\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-zbn6s"
Oct 30 21:08:40.881509 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:08:40.881479    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fc825c3-c94c-452c-9cfe-96907a565821-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-zbn6s\" (UID: \"9fc825c3-c94c-452c-9cfe-96907a565821\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-zbn6s"
Oct 30 21:08:40.883282 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:08:40.883260    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fc825c3-c94c-452c-9cfe-96907a565821-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-zbn6s\" (UID: \"9fc825c3-c94c-452c-9cfe-96907a565821\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-zbn6s"
Oct 30 21:08:40.893818 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:08:40.893793    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmzbw\" (UniqueName: \"kubernetes.io/projected/9fc825c3-c94c-452c-9cfe-96907a565821-kube-api-access-cmzbw\") pod \"maas-api-revoke-sub-e2fc9c5b-zbn6s\" (UID: \"9fc825c3-c94c-452c-9cfe-96907a565821\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-zbn6s"
Oct 30 21:08:41.034719 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:08:41.034630    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-zbn6s"
Oct 30 21:08:41.312524 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:08:41.312497    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-zbn6s"]
Oct 30 21:08:41.314166 ip-10-0-131-125 kubenswrapper[4521]: W1030 21:08:41.314133    4521 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fc825c3_c94c_452c_9cfe_96907a565821.slice/crio-1177e697cd2d1b299edaf2ac7e59ace27c0e087f756a43d483379b1a53603f2e WatchSource:0}: Error finding container 1177e697cd2d1b299edaf2ac7e59ace27c0e087f756a43d483379b1a53603f2e: Status 404 returned error can't find the container with id 1177e697cd2d1b299edaf2ac7e59ace27c0e087f756a43d483379b1a53603f2e
Oct 30 21:08:42.165627 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:08:42.165591    4521 generic.go:356] "Generic (PLEG): container finished" podID="9fc825c3-c94c-452c-9cfe-96907a565821" containerID="2fb97a194a31e2ebdce192551e07a627ec78921de418b315c4ebd2603af9d7ae" exitCode=0
Oct 30 21:08:42.166040 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:08:42.165658    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-zbn6s" event={"ID":"9fc825c3-c94c-452c-9cfe-96907a565821","Type":"ContainerDied","Data":"2fb97a194a31e2ebdce192551e07a627ec78921de418b315c4ebd2603af9d7ae"}
Oct 30 21:08:42.166040 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:08:42.165696    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-zbn6s" event={"ID":"9fc825c3-c94c-452c-9cfe-96907a565821","Type":"ContainerStarted","Data":"1177e697cd2d1b299edaf2ac7e59ace27c0e087f756a43d483379b1a53603f2e"}
Oct 30 21:08:43.299622 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:08:43.299596    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-zbn6s"
Oct 30 21:08:43.403792 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:08:43.403751    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9fc825c3-c94c-452c-9cfe-96907a565821-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fc825c3-c94c-452c-9cfe-96907a565821-maas-api-xks-ca\") pod \"9fc825c3-c94c-452c-9cfe-96907a565821\" (UID: \"9fc825c3-c94c-452c-9cfe-96907a565821\") "
Oct 30 21:08:43.404001 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:08:43.403821    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fc825c3-c94c-452c-9cfe-96907a565821-kube-api-access-cmzbw\" (UniqueName: \"kubernetes.io/projected/9fc825c3-c94c-452c-9cfe-96907a565821-kube-api-access-cmzbw\") pod \"9fc825c3-c94c-452c-9cfe-96907a565821\" (UID: \"9fc825c3-c94c-452c-9cfe-96907a565821\") "
Oct 30 21:08:43.404001 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:08:43.403899    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9fc825c3-c94c-452c-9cfe-96907a565821-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fc825c3-c94c-452c-9cfe-96907a565821-maas-api-openshift-ca\") pod \"9fc825c3-c94c-452c-9cfe-96907a565821\" (UID: \"9fc825c3-c94c-452c-9cfe-96907a565821\") "
Oct 30 21:08:43.404276 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:08:43.404252    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fc825c3-c94c-452c-9cfe-96907a565821-maas-api-openshift-ca" pod "9fc825c3-c94c-452c-9cfe-96907a565821" (UID: "9fc825c3-c94c-452c-9cfe-96907a565821"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:08:43.406015 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:08:43.405985    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fc825c3-c94c-452c-9cfe-96907a565821-maas-api-xks-ca" pod "9fc825c3-c94c-452c-9cfe-96907a565821" (UID: "9fc825c3-c94c-452c-9cfe-96907a565821"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:08:43.406117 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:08:43.406010    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fc825c3-c94c-452c-9cfe-96907a565821-kube-api-access-cmzbw" pod "9fc825c3-c94c-452c-9cfe-96907a565821" (UID: "9fc825c3-c94c-452c-9cfe-96907a565821"). InnerVolumeSpecName "kube-api-access-cmzbw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:08:43.504787 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:08:43.504750    4521 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fc825c3-c94c-452c-9cfe-96907a565821-maas-api-openshift-ca\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:08:43.504787 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:08:43.504782    4521 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fc825c3-c94c-452c-9cfe-96907a565821-maas-api-xks-ca\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:08:43.504787 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:08:43.504794    4521 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmzbw\" (UniqueName: \"kubernetes.io/projected/9fc825c3-c94c-452c-9cfe-96907a565821-kube-api-access-cmzbw\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:08:44.178723 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:08:44.178686    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-zbn6s"
Oct 30 21:08:44.178909 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:08:44.178719    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-zbn6s" event={"ID":"9fc825c3-c94c-452c-9cfe-96907a565821","Type":"ContainerDied","Data":"1177e697cd2d1b299edaf2ac7e59ace27c0e087f756a43d483379b1a53603f2e"}
Oct 30 21:08:44.178909 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:08:44.178765    4521 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1177e697cd2d1b299edaf2ac7e59ace27c0e087f756a43d483379b1a53603f2e"
Oct 30 21:08:46.427984 ip-10-0-131-125 kubenswrapper[4521]: E1030 21:08:46.427946    4521 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lkn2k" containerName="extractor"
Oct 30 21:08:50.937263 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:08:50.937227    4521 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xhgmn"]
Oct 30 21:08:50.937670 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:08:50.937626    4521 cpu_manager.go:414] "containerMap: removing container" podUID="9fc825c3-c94c-452c-9cfe-96907a565821" containerName="revoke-keys"
Oct 30 21:08:50.937670 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:08:50.937638    4521 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fc825c3-c94c-452c-9cfe-96907a565821" containerName="revoke-keys"
Oct 30 21:08:50.937740 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:08:50.937694    4521 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fc825c3-c94c-452c-9cfe-96907a565821" containerName="revoke-keys"
Oct 30 21:08:50.940749 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:08:50.940723    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xhgmn"
Oct 30 21:08:50.944788 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:08:50.944763    4521 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:08:50.945930 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:08:50.945907    4521 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:08:50.946053 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:08:50.945907    4521 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:08:50.946053 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:08:50.945941    4521 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c45pw\""
Oct 30 21:08:50.953894 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:08:50.953132    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xhgmn"]
Oct 30 21:08:51.073801 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:08:51.073760    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/775d6d2d-e3eb-45b6-aeae-028f211a1f74-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-xhgmn\" (UID: \"775d6d2d-e3eb-45b6-aeae-028f211a1f74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xhgmn"
Oct 30 21:08:51.073993 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:08:51.073818    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/775d6d2d-e3eb-45b6-aeae-028f211a1f74-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-xhgmn\" (UID: \"775d6d2d-e3eb-45b6-aeae-028f211a1f74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xhgmn"
Oct 30 21:08:51.073993 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:08:51.073903    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lnpq\" (UniqueName: \"kubernetes.io/projected/775d6d2d-e3eb-45b6-aeae-028f211a1f74-kube-api-access-5lnpq\") pod \"maas-api-revoke-sub-5a29efb3-xhgmn\" (UID: \"775d6d2d-e3eb-45b6-aeae-028f211a1f74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xhgmn"
Oct 30 21:08:51.175436 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:08:51.175396    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/775d6d2d-e3eb-45b6-aeae-028f211a1f74-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-xhgmn\" (UID: \"775d6d2d-e3eb-45b6-aeae-028f211a1f74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xhgmn"
Oct 30 21:08:51.175771 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:08:51.175449    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/775d6d2d-e3eb-45b6-aeae-028f211a1f74-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-xhgmn\" (UID: \"775d6d2d-e3eb-45b6-aeae-028f211a1f74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xhgmn"
Oct 30 21:08:51.175771 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:08:51.175476    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lnpq\" (UniqueName: \"kubernetes.io/projected/775d6d2d-e3eb-45b6-aeae-028f211a1f74-kube-api-access-5lnpq\") pod \"maas-api-revoke-sub-5a29efb3-xhgmn\" (UID: \"775d6d2d-e3eb-45b6-aeae-028f211a1f74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xhgmn"
Oct 30 21:08:51.176107 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:08:51.176083    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/775d6d2d-e3eb-45b6-aeae-028f211a1f74-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-xhgmn\" (UID: \"775d6d2d-e3eb-45b6-aeae-028f211a1f74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xhgmn"
Oct 30 21:08:51.177919 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:08:51.177894    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/775d6d2d-e3eb-45b6-aeae-028f211a1f74-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-xhgmn\" (UID: \"775d6d2d-e3eb-45b6-aeae-028f211a1f74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xhgmn"
Oct 30 21:08:51.188003 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:08:51.187944    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lnpq\" (UniqueName: \"kubernetes.io/projected/775d6d2d-e3eb-45b6-aeae-028f211a1f74-kube-api-access-5lnpq\") pod \"maas-api-revoke-sub-5a29efb3-xhgmn\" (UID: \"775d6d2d-e3eb-45b6-aeae-028f211a1f74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xhgmn"
Oct 30 21:08:51.259030 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:08:51.258987    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xhgmn"
Oct 30 21:08:51.540352 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:08:51.540305    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xhgmn"]
Oct 30 21:08:51.540885 ip-10-0-131-125 kubenswrapper[4521]: W1030 21:08:51.540850    4521 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod775d6d2d_e3eb_45b6_aeae_028f211a1f74.slice/crio-c763bbba5655432f79b43311a563e5d6e81a6d427dfdf30234457e46917e7571 WatchSource:0}: Error finding container c763bbba5655432f79b43311a563e5d6e81a6d427dfdf30234457e46917e7571: Status 404 returned error can't find the container with id c763bbba5655432f79b43311a563e5d6e81a6d427dfdf30234457e46917e7571
Oct 30 21:08:51.702944 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:08:51.702911    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4kxb"]
Oct 30 21:08:52.228859 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:08:52.228808    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xhgmn" event={"ID":"775d6d2d-e3eb-45b6-aeae-028f211a1f74","Type":"ContainerStarted","Data":"739064a2d8a063d5dc9424bfc3975a6e3a8c668e4ae149dec50cbc2b43e66b8c"}
Oct 30 21:08:52.229390 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:08:52.228882    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xhgmn" event={"ID":"775d6d2d-e3eb-45b6-aeae-028f211a1f74","Type":"ContainerStarted","Data":"c763bbba5655432f79b43311a563e5d6e81a6d427dfdf30234457e46917e7571"}
Oct 30 21:08:52.248418 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:08:52.248352    4521 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xhgmn" podStartSLOduration=2.248314617 podStartE2EDuration="2.248314617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:08:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:08:52.24800718 +0000 UTC m=+1799.593628420" watchObservedRunningTime="2026-10-30 21:08:52.248314617 +0000 UTC m=+1799.593935856"
Oct 30 21:08:53.279349 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:08:53.278370    4521 generic.go:356] "Generic (PLEG): container finished" podID="775d6d2d-e3eb-45b6-aeae-028f211a1f74" containerID="739064a2d8a063d5dc9424bfc3975a6e3a8c668e4ae149dec50cbc2b43e66b8c" exitCode=0
Oct 30 21:08:53.279349 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:08:53.278518    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xhgmn" event={"ID":"775d6d2d-e3eb-45b6-aeae-028f211a1f74","Type":"ContainerDied","Data":"739064a2d8a063d5dc9424bfc3975a6e3a8c668e4ae149dec50cbc2b43e66b8c"}
Oct 30 21:08:53.690566 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:08:53.690536    4521 scope.go:122] "RemoveContainer" containerID="a8372a359a760d7091e41da2dfbfdc93b44450e5db6246bbcba52fd64c2473b0"
Oct 30 21:08:53.709065 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:08:53.708969    4521 scope.go:122] "RemoveContainer" containerID="8a9c2791366a819fceeb64882405b3c156aa337e0b21f32cf95fdc861eefc465"
Oct 30 21:08:53.785969 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:08:53.785827    4521 scope.go:122] "RemoveContainer" containerID="5cf7f7a8b1f335911c58c99ac404c84165377fe9f2d6e87979e4ec7dc0d1b324"
Oct 30 21:08:53.805923 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:08:53.805890    4521 scope.go:122] "RemoveContainer" containerID="9fd1240d2ea9a4377d88ba3c064b8ee0e27bbf31b95caeafb9ccf062c6800b7f"
Oct 30 21:08:53.840473 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:08:53.840127    4521 scope.go:122] "RemoveContainer" containerID="9e011a83e7dad15b724390446e366ea7ab9f3a5ba297591c56164b3b6ad78ab2"
Oct 30 21:08:53.889651 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:08:53.889510    4521 scope.go:122] "RemoveContainer" containerID="89b11ef9292b842b495a43808bec52f06d5dcca68b1dcb3612d604c6d4314d37"
Oct 30 21:08:53.901571 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:08:53.901554    4521 scope.go:122] "RemoveContainer" containerID="c9b38afcfda0af7dba2226ce02ebc4ca7a37be554534fdfe24d8bb9587f744e3"
Oct 30 21:08:53.972359 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:08:53.962709    4521 scope.go:122] "RemoveContainer" containerID="99129f8e7124d04a27ac49c5fe925c7dbcb4e81dca8a1358e7d504d0b18a897d"
Oct 30 21:08:53.996312 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:08:53.996289    4521 scope.go:122] "RemoveContainer" containerID="25ed489b32819722fcf37e60849dffcb8c1c0160dc8b3ead209e18907d7904b5"
Oct 30 21:08:54.005292 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:08:54.005269    4521 scope.go:122] "RemoveContainer" containerID="2902133771aa127fda482d8b7422a5e0b96f65de64b58535eae12ed27c79d77a"
Oct 30 21:08:54.053668 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:08:54.052255    4521 scope.go:122] "RemoveContainer" containerID="e80142b954d75601b661e585265be56c91785e7ddf5825513b4e605e0e09c3d8"
Oct 30 21:08:54.098619 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:08:54.098495    4521 scope.go:122] "RemoveContainer" containerID="47e96b2aa0754a22e6b855b58362b8058daa069b9da3e86fce6eda12764f3c7d"
Oct 30 21:08:54.108993 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:08:54.108948    4521 scope.go:122] "RemoveContainer" containerID="d4834ace6cb1b60c0405fe5ae3fe13c0d4ef269177a8799eb4f2bd94535177c6"
Oct 30 21:08:54.174453 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:08:54.167526    4521 scope.go:122] "RemoveContainer" containerID="d789c2cd2837495b6fc2814d9d24c7e5369b4d7973a3018bbe7519b278dd793b"
Oct 30 21:08:54.717792 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:08:54.717761    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xhgmn"
Oct 30 21:08:54.801826 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:08:54.801741    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/775d6d2d-e3eb-45b6-aeae-028f211a1f74-kube-api-access-5lnpq\" (UniqueName: \"kubernetes.io/projected/775d6d2d-e3eb-45b6-aeae-028f211a1f74-kube-api-access-5lnpq\") pod \"775d6d2d-e3eb-45b6-aeae-028f211a1f74\" (UID: \"775d6d2d-e3eb-45b6-aeae-028f211a1f74\") "
Oct 30 21:08:54.801826 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:08:54.801793    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/775d6d2d-e3eb-45b6-aeae-028f211a1f74-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/775d6d2d-e3eb-45b6-aeae-028f211a1f74-maas-api-xks-ca\") pod \"775d6d2d-e3eb-45b6-aeae-028f211a1f74\" (UID: \"775d6d2d-e3eb-45b6-aeae-028f211a1f74\") "
Oct 30 21:08:54.802063 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:08:54.801878    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/775d6d2d-e3eb-45b6-aeae-028f211a1f74-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/775d6d2d-e3eb-45b6-aeae-028f211a1f74-maas-api-openshift-ca\") pod \"775d6d2d-e3eb-45b6-aeae-028f211a1f74\" (UID: \"775d6d2d-e3eb-45b6-aeae-028f211a1f74\") "
Oct 30 21:08:54.802243 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:08:54.802210    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/775d6d2d-e3eb-45b6-aeae-028f211a1f74-maas-api-openshift-ca" pod "775d6d2d-e3eb-45b6-aeae-028f211a1f74" (UID: "775d6d2d-e3eb-45b6-aeae-028f211a1f74"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:08:54.803891 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:08:54.803869    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/775d6d2d-e3eb-45b6-aeae-028f211a1f74-maas-api-xks-ca" pod "775d6d2d-e3eb-45b6-aeae-028f211a1f74" (UID: "775d6d2d-e3eb-45b6-aeae-028f211a1f74"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:08:54.803989 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:08:54.803966    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/775d6d2d-e3eb-45b6-aeae-028f211a1f74-kube-api-access-5lnpq" pod "775d6d2d-e3eb-45b6-aeae-028f211a1f74" (UID: "775d6d2d-e3eb-45b6-aeae-028f211a1f74"). InnerVolumeSpecName "kube-api-access-5lnpq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:08:54.903504 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:08:54.903466    4521 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/775d6d2d-e3eb-45b6-aeae-028f211a1f74-maas-api-openshift-ca\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:08:54.903504 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:08:54.903500    4521 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5lnpq\" (UniqueName: \"kubernetes.io/projected/775d6d2d-e3eb-45b6-aeae-028f211a1f74-kube-api-access-5lnpq\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:08:54.903725 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:08:54.903516    4521 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/775d6d2d-e3eb-45b6-aeae-028f211a1f74-maas-api-xks-ca\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:08:55.355297 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:08:55.355265    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xhgmn"
Oct 30 21:08:55.355297 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:08:55.355293    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xhgmn" event={"ID":"775d6d2d-e3eb-45b6-aeae-028f211a1f74","Type":"ContainerDied","Data":"c763bbba5655432f79b43311a563e5d6e81a6d427dfdf30234457e46917e7571"}
Oct 30 21:08:55.355522 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:08:55.355353    4521 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c763bbba5655432f79b43311a563e5d6e81a6d427dfdf30234457e46917e7571"
Oct 30 21:08:55.383156 ip-10-0-131-125 kubenswrapper[4521]: E1030 21:08:55.383125    4521 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod775d6d2d_e3eb_45b6_aeae_028f211a1f74.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod775d6d2d_e3eb_45b6_aeae_028f211a1f74.slice/crio-c763bbba5655432f79b43311a563e5d6e81a6d427dfdf30234457e46917e7571\": RecentStats: unable to find data in memory cache]"
Oct 30 21:09:01.722595 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:09:01.722555    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4kxb"]
Oct 30 21:09:16.428718 ip-10-0-131-125 kubenswrapper[4521]: E1030 21:09:16.428680    4521 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lkn2k" containerName="extractor"
Oct 30 21:09:46.429338 ip-10-0-131-125 kubenswrapper[4521]: E1030 21:09:46.429285    4521 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lkn2k" containerName="extractor"
Oct 30 21:09:54.607270 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:09:54.607240    4521 scope.go:122] "RemoveContainer" containerID="ec55a4368f21b5a8515dc4b6387e36f645a1105661ae41d3f2ae7d16d642c0c1"
Oct 30 21:10:16.427850 ip-10-0-131-125 kubenswrapper[4521]: E1030 21:10:16.427813    4521 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lkn2k" containerName="extractor"
Oct 30 21:10:46.428621 ip-10-0-131-125 kubenswrapper[4521]: E1030 21:10:46.428577    4521 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lkn2k" containerName="extractor"
Oct 30 21:11:16.428752 ip-10-0-131-125 kubenswrapper[4521]: E1030 21:11:16.428667    4521 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lkn2k" containerName="extractor"
Oct 30 21:11:46.429602 ip-10-0-131-125 kubenswrapper[4521]: E1030 21:11:46.429565    4521 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lkn2k" containerName="extractor"
Oct 30 21:12:16.428459 ip-10-0-131-125 kubenswrapper[4521]: E1030 21:12:16.428415    4521 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lkn2k" containerName="extractor"
Oct 30 21:12:46.428852 ip-10-0-131-125 kubenswrapper[4521]: E1030 21:12:46.428809    4521 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lkn2k" containerName="extractor"
Oct 30 21:13:16.427800 ip-10-0-131-125 kubenswrapper[4521]: E1030 21:13:16.427762    4521 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lkn2k" containerName="extractor"
Oct 30 21:13:44.036564 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:13:44.036525    4521 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-zbn6s"]
Oct 30 21:13:44.040430 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:13:44.040403    4521 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-zbn6s"]
Oct 30 21:13:44.791227 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:13:44.791183    4521 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9fc825c3-c94c-452c-9cfe-96907a565821" path="/var/lib/kubelet/pods/9fc825c3-c94c-452c-9cfe-96907a565821/volumes"
Oct 30 21:13:46.428677 ip-10-0-131-125 kubenswrapper[4521]: E1030 21:13:46.428639    4521 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lkn2k" containerName="extractor"
Oct 30 21:13:55.021917 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:13:55.021879    4521 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xhgmn"]
Oct 30 21:13:55.024919 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:13:55.024892    4521 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xhgmn"]
Oct 30 21:13:56.791213 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:13:56.791178    4521 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="775d6d2d-e3eb-45b6-aeae-028f211a1f74" path="/var/lib/kubelet/pods/775d6d2d-e3eb-45b6-aeae-028f211a1f74/volumes"
Oct 30 21:14:16.428995 ip-10-0-131-125 kubenswrapper[4521]: E1030 21:14:16.428915    4521 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lkn2k" containerName="extractor"
Oct 30 21:14:20.411199 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:14:20.411163    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4kxb"]
Oct 30 21:14:24.415498 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:14:24.415454    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4kxb"]
Oct 30 21:14:25.926064 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:14:25.926024    4521 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-sq68x"]
Oct 30 21:14:25.926664 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:14:25.926284    4521 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-sq68x" podUID="16e33702-94a7-4c37-b430-1ac4f002832b" containerName="istio-proxy" containerID="cri-o://7c80e51bd8781a3c0b20d8fa3f7b9051d5c61245c2928324f61e13ad1d9f0cb3" gracePeriod=30
Oct 30 21:14:25.933885 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:14:25.933854    4521 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-54dql"]
Oct 30 21:14:25.934103 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:14:25.934080    4521 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-54dql" podUID="be342bb0-ebf7-4bb3-a929-5abfc455531e" containerName="istio-proxy" containerID="cri-o://ad973049c03a72fe30ab539571ece3e2307fd57f74e16ecc8dfb1a5eb2194e13" gracePeriod=30
Oct 30 21:14:31.078264 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:14:31.078227    4521 generic.go:356] "Generic (PLEG): container finished" podID="be342bb0-ebf7-4bb3-a929-5abfc455531e" containerID="ad973049c03a72fe30ab539571ece3e2307fd57f74e16ecc8dfb1a5eb2194e13" exitCode=0
Oct 30 21:14:31.078807 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:14:31.078386    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-54dql" event={"ID":"be342bb0-ebf7-4bb3-a929-5abfc455531e","Type":"ContainerDied","Data":"ad973049c03a72fe30ab539571ece3e2307fd57f74e16ecc8dfb1a5eb2194e13"}
Oct 30 21:14:31.080244 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:14:31.080151    4521 generic.go:356] "Generic (PLEG): container finished" podID="16e33702-94a7-4c37-b430-1ac4f002832b" containerID="7c80e51bd8781a3c0b20d8fa3f7b9051d5c61245c2928324f61e13ad1d9f0cb3" exitCode=0
Oct 30 21:14:31.080244 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:14:31.080189    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-sq68x" event={"ID":"16e33702-94a7-4c37-b430-1ac4f002832b","Type":"ContainerDied","Data":"7c80e51bd8781a3c0b20d8fa3f7b9051d5c61245c2928324f61e13ad1d9f0cb3"}
Oct 30 21:14:31.178631 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:14:31.178600    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-sq68x"
Oct 30 21:14:31.214436 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:14:31.214364    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-54dql"
Oct 30 21:14:31.274662 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:14:31.274632    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be342bb0-ebf7-4bb3-a929-5abfc455531e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be342bb0-ebf7-4bb3-a929-5abfc455531e-istiod-ca-cert\") pod \"be342bb0-ebf7-4bb3-a929-5abfc455531e\" (UID: \"be342bb0-ebf7-4bb3-a929-5abfc455531e\") "
Oct 30 21:14:31.274841 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:14:31.274685    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be342bb0-ebf7-4bb3-a929-5abfc455531e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be342bb0-ebf7-4bb3-a929-5abfc455531e-istio-data\") pod \"be342bb0-ebf7-4bb3-a929-5abfc455531e\" (UID: \"be342bb0-ebf7-4bb3-a929-5abfc455531e\") "
Oct 30 21:14:31.274841 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:14:31.274711    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16e33702-94a7-4c37-b430-1ac4f002832b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16e33702-94a7-4c37-b430-1ac4f002832b-workload-certs\") pod \"16e33702-94a7-4c37-b430-1ac4f002832b\" (UID: \"16e33702-94a7-4c37-b430-1ac4f002832b\") "
Oct 30 21:14:31.274841 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:14:31.274733    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16e33702-94a7-4c37-b430-1ac4f002832b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16e33702-94a7-4c37-b430-1ac4f002832b-credential-socket\") pod \"16e33702-94a7-4c37-b430-1ac4f002832b\" (UID: \"16e33702-94a7-4c37-b430-1ac4f002832b\") "
Oct 30 21:14:31.274841 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:14:31.274776    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16e33702-94a7-4c37-b430-1ac4f002832b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16e33702-94a7-4c37-b430-1ac4f002832b-istio-data\") pod \"16e33702-94a7-4c37-b430-1ac4f002832b\" (UID: \"16e33702-94a7-4c37-b430-1ac4f002832b\") "
Oct 30 21:14:31.274841 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:14:31.274804    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/16e33702-94a7-4c37-b430-1ac4f002832b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16e33702-94a7-4c37-b430-1ac4f002832b-istio-podinfo\") pod \"16e33702-94a7-4c37-b430-1ac4f002832b\" (UID: \"16e33702-94a7-4c37-b430-1ac4f002832b\") "
Oct 30 21:14:31.275126 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:14:31.274867    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be342bb0-ebf7-4bb3-a929-5abfc455531e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be342bb0-ebf7-4bb3-a929-5abfc455531e-workload-socket\") pod \"be342bb0-ebf7-4bb3-a929-5abfc455531e\" (UID: \"be342bb0-ebf7-4bb3-a929-5abfc455531e\") "
Oct 30 21:14:31.275126 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:14:31.274894    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be342bb0-ebf7-4bb3-a929-5abfc455531e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be342bb0-ebf7-4bb3-a929-5abfc455531e-istio-envoy\") pod \"be342bb0-ebf7-4bb3-a929-5abfc455531e\" (UID: \"be342bb0-ebf7-4bb3-a929-5abfc455531e\") "
Oct 30 21:14:31.275126 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:14:31.274925    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be342bb0-ebf7-4bb3-a929-5abfc455531e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be342bb0-ebf7-4bb3-a929-5abfc455531e-credential-socket\") pod \"be342bb0-ebf7-4bb3-a929-5abfc455531e\" (UID: \"be342bb0-ebf7-4bb3-a929-5abfc455531e\") "
Oct 30 21:14:31.275126 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:14:31.274962    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16e33702-94a7-4c37-b430-1ac4f002832b-kube-api-access-vhqf5\" (UniqueName: \"kubernetes.io/projected/16e33702-94a7-4c37-b430-1ac4f002832b-kube-api-access-vhqf5\") pod \"16e33702-94a7-4c37-b430-1ac4f002832b\" (UID: \"16e33702-94a7-4c37-b430-1ac4f002832b\") "
Oct 30 21:14:31.275126 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:14:31.274998    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16e33702-94a7-4c37-b430-1ac4f002832b-istio-token\" (UniqueName: \"kubernetes.io/projected/16e33702-94a7-4c37-b430-1ac4f002832b-istio-token\") pod \"16e33702-94a7-4c37-b430-1ac4f002832b\" (UID: \"16e33702-94a7-4c37-b430-1ac4f002832b\") "
Oct 30 21:14:31.275126 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:14:31.275032    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be342bb0-ebf7-4bb3-a929-5abfc455531e-istio-token\" (UniqueName: \"kubernetes.io/projected/be342bb0-ebf7-4bb3-a929-5abfc455531e-istio-token\") pod \"be342bb0-ebf7-4bb3-a929-5abfc455531e\" (UID: \"be342bb0-ebf7-4bb3-a929-5abfc455531e\") "
Oct 30 21:14:31.275126 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:14:31.275050    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16e33702-94a7-4c37-b430-1ac4f002832b-credential-socket" pod "16e33702-94a7-4c37-b430-1ac4f002832b" (UID: "16e33702-94a7-4c37-b430-1ac4f002832b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:14:31.275126 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:14:31.275061    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16e33702-94a7-4c37-b430-1ac4f002832b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16e33702-94a7-4c37-b430-1ac4f002832b-istio-envoy\") pod \"16e33702-94a7-4c37-b430-1ac4f002832b\" (UID: \"16e33702-94a7-4c37-b430-1ac4f002832b\") "
Oct 30 21:14:31.275126 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:14:31.275057    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be342bb0-ebf7-4bb3-a929-5abfc455531e-istio-data" pod "be342bb0-ebf7-4bb3-a929-5abfc455531e" (UID: "be342bb0-ebf7-4bb3-a929-5abfc455531e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:14:31.275126 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:14:31.275095    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16e33702-94a7-4c37-b430-1ac4f002832b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16e33702-94a7-4c37-b430-1ac4f002832b-workload-socket\") pod \"16e33702-94a7-4c37-b430-1ac4f002832b\" (UID: \"16e33702-94a7-4c37-b430-1ac4f002832b\") "
Oct 30 21:14:31.275126 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:14:31.275098    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16e33702-94a7-4c37-b430-1ac4f002832b-workload-certs" pod "16e33702-94a7-4c37-b430-1ac4f002832b" (UID: "16e33702-94a7-4c37-b430-1ac4f002832b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:14:31.275126 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:14:31.275123    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be342bb0-ebf7-4bb3-a929-5abfc455531e-istiod-ca-cert" pod "be342bb0-ebf7-4bb3-a929-5abfc455531e" (UID: "be342bb0-ebf7-4bb3-a929-5abfc455531e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:14:31.275126 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:14:31.275131    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16e33702-94a7-4c37-b430-1ac4f002832b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16e33702-94a7-4c37-b430-1ac4f002832b-istiod-ca-cert\") pod \"16e33702-94a7-4c37-b430-1ac4f002832b\" (UID: \"16e33702-94a7-4c37-b430-1ac4f002832b\") "
Oct 30 21:14:31.275834 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:14:31.275172    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be342bb0-ebf7-4bb3-a929-5abfc455531e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be342bb0-ebf7-4bb3-a929-5abfc455531e-workload-certs\") pod \"be342bb0-ebf7-4bb3-a929-5abfc455531e\" (UID: \"be342bb0-ebf7-4bb3-a929-5abfc455531e\") "
Oct 30 21:14:31.275834 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:14:31.275214    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be342bb0-ebf7-4bb3-a929-5abfc455531e-kube-api-access-xh4d2\" (UniqueName: \"kubernetes.io/projected/be342bb0-ebf7-4bb3-a929-5abfc455531e-kube-api-access-xh4d2\") pod \"be342bb0-ebf7-4bb3-a929-5abfc455531e\" (UID: \"be342bb0-ebf7-4bb3-a929-5abfc455531e\") "
Oct 30 21:14:31.275834 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:14:31.275247    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/be342bb0-ebf7-4bb3-a929-5abfc455531e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be342bb0-ebf7-4bb3-a929-5abfc455531e-istio-podinfo\") pod \"be342bb0-ebf7-4bb3-a929-5abfc455531e\" (UID: \"be342bb0-ebf7-4bb3-a929-5abfc455531e\") "
Oct 30 21:14:31.275834 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:14:31.275359    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be342bb0-ebf7-4bb3-a929-5abfc455531e-credential-socket" pod "be342bb0-ebf7-4bb3-a929-5abfc455531e" (UID: "be342bb0-ebf7-4bb3-a929-5abfc455531e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:14:31.275834 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:14:31.275586    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16e33702-94a7-4c37-b430-1ac4f002832b-istiod-ca-cert" pod "16e33702-94a7-4c37-b430-1ac4f002832b" (UID: "16e33702-94a7-4c37-b430-1ac4f002832b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:14:31.275834 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:14:31.275617    4521 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16e33702-94a7-4c37-b430-1ac4f002832b-workload-certs\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:14:31.275834 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:14:31.275639    4521 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16e33702-94a7-4c37-b430-1ac4f002832b-credential-socket\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:14:31.275834 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:14:31.275654    4521 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be342bb0-ebf7-4bb3-a929-5abfc455531e-credential-socket\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:14:31.275834 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:14:31.275668    4521 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be342bb0-ebf7-4bb3-a929-5abfc455531e-istiod-ca-cert\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:14:31.275834 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:14:31.275685    4521 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be342bb0-ebf7-4bb3-a929-5abfc455531e-istio-data\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:14:31.276509 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:14:31.275946    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be342bb0-ebf7-4bb3-a929-5abfc455531e-workload-certs" pod "be342bb0-ebf7-4bb3-a929-5abfc455531e" (UID: "be342bb0-ebf7-4bb3-a929-5abfc455531e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:14:31.276509 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:14:31.276050    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16e33702-94a7-4c37-b430-1ac4f002832b-istio-data" pod "16e33702-94a7-4c37-b430-1ac4f002832b" (UID: "16e33702-94a7-4c37-b430-1ac4f002832b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:14:31.276720 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:14:31.276696    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16e33702-94a7-4c37-b430-1ac4f002832b-workload-socket" pod "16e33702-94a7-4c37-b430-1ac4f002832b" (UID: "16e33702-94a7-4c37-b430-1ac4f002832b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:14:31.276789 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:14:31.276745    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be342bb0-ebf7-4bb3-a929-5abfc455531e-workload-socket" pod "be342bb0-ebf7-4bb3-a929-5abfc455531e" (UID: "be342bb0-ebf7-4bb3-a929-5abfc455531e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:14:31.278777 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:14:31.278749    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16e33702-94a7-4c37-b430-1ac4f002832b-kube-api-access-vhqf5" pod "16e33702-94a7-4c37-b430-1ac4f002832b" (UID: "16e33702-94a7-4c37-b430-1ac4f002832b"). InnerVolumeSpecName "kube-api-access-vhqf5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:14:31.278777 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:14:31.278762    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16e33702-94a7-4c37-b430-1ac4f002832b-istio-token" pod "16e33702-94a7-4c37-b430-1ac4f002832b" (UID: "16e33702-94a7-4c37-b430-1ac4f002832b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:14:31.278955 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:14:31.278772    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/16e33702-94a7-4c37-b430-1ac4f002832b-istio-podinfo" pod "16e33702-94a7-4c37-b430-1ac4f002832b" (UID: "16e33702-94a7-4c37-b430-1ac4f002832b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:14:31.278955 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:14:31.278869    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be342bb0-ebf7-4bb3-a929-5abfc455531e-istio-envoy" pod "be342bb0-ebf7-4bb3-a929-5abfc455531e" (UID: "be342bb0-ebf7-4bb3-a929-5abfc455531e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:14:31.278955 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:14:31.278890    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/be342bb0-ebf7-4bb3-a929-5abfc455531e-istio-podinfo" pod "be342bb0-ebf7-4bb3-a929-5abfc455531e" (UID: "be342bb0-ebf7-4bb3-a929-5abfc455531e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:14:31.279121 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:14:31.279005    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16e33702-94a7-4c37-b430-1ac4f002832b-istio-envoy" pod "16e33702-94a7-4c37-b430-1ac4f002832b" (UID: "16e33702-94a7-4c37-b430-1ac4f002832b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:14:31.279211 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:14:31.279196    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be342bb0-ebf7-4bb3-a929-5abfc455531e-kube-api-access-xh4d2" pod "be342bb0-ebf7-4bb3-a929-5abfc455531e" (UID: "be342bb0-ebf7-4bb3-a929-5abfc455531e"). InnerVolumeSpecName "kube-api-access-xh4d2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:14:31.280546 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:14:31.280526    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be342bb0-ebf7-4bb3-a929-5abfc455531e-istio-token" pod "be342bb0-ebf7-4bb3-a929-5abfc455531e" (UID: "be342bb0-ebf7-4bb3-a929-5abfc455531e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:14:31.376588 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:14:31.376533    4521 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16e33702-94a7-4c37-b430-1ac4f002832b-istio-data\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:14:31.376588 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:14:31.376590    4521 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16e33702-94a7-4c37-b430-1ac4f002832b-istio-podinfo\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:14:31.376795 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:14:31.376605    4521 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be342bb0-ebf7-4bb3-a929-5abfc455531e-workload-socket\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:14:31.376795 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:14:31.376619    4521 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be342bb0-ebf7-4bb3-a929-5abfc455531e-istio-envoy\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:14:31.376795 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:14:31.376633    4521 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhqf5\" (UniqueName: \"kubernetes.io/projected/16e33702-94a7-4c37-b430-1ac4f002832b-kube-api-access-vhqf5\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:14:31.376795 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:14:31.376647    4521 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16e33702-94a7-4c37-b430-1ac4f002832b-istio-token\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:14:31.376795 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:14:31.376660    4521 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be342bb0-ebf7-4bb3-a929-5abfc455531e-istio-token\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:14:31.376795 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:14:31.376674    4521 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16e33702-94a7-4c37-b430-1ac4f002832b-istio-envoy\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:14:31.376795 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:14:31.376688    4521 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16e33702-94a7-4c37-b430-1ac4f002832b-workload-socket\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:14:31.376795 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:14:31.376700    4521 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16e33702-94a7-4c37-b430-1ac4f002832b-istiod-ca-cert\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:14:31.376795 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:14:31.376713    4521 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be342bb0-ebf7-4bb3-a929-5abfc455531e-workload-certs\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:14:31.376795 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:14:31.376726    4521 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xh4d2\" (UniqueName: \"kubernetes.io/projected/be342bb0-ebf7-4bb3-a929-5abfc455531e-kube-api-access-xh4d2\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:14:31.376795 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:14:31.376739    4521 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be342bb0-ebf7-4bb3-a929-5abfc455531e-istio-podinfo\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:14:32.086632 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:14:32.086600    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-54dql"
Oct 30 21:14:32.087135 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:14:32.086626    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-54dql" event={"ID":"be342bb0-ebf7-4bb3-a929-5abfc455531e","Type":"ContainerDied","Data":"0f105deb16b2be962734237f2dddbd2b0e5a5d2533aadfb0d9ed6038e133c358"}
Oct 30 21:14:32.087135 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:14:32.086688    4521 scope.go:122] "RemoveContainer" containerID="ad973049c03a72fe30ab539571ece3e2307fd57f74e16ecc8dfb1a5eb2194e13"
Oct 30 21:14:32.088134 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:14:32.088089    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-sq68x" event={"ID":"16e33702-94a7-4c37-b430-1ac4f002832b","Type":"ContainerDied","Data":"2872e5bb18477333a50df4e0c546e4de1fa65b592daa6039c94435059cccbc08"}
Oct 30 21:14:32.088219 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:14:32.088148    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-sq68x"
Oct 30 21:14:32.096785 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:14:32.096765    4521 scope.go:122] "RemoveContainer" containerID="7c80e51bd8781a3c0b20d8fa3f7b9051d5c61245c2928324f61e13ad1d9f0cb3"
Oct 30 21:14:32.115786 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:14:32.115747    4521 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-54dql"]
Oct 30 21:14:32.117882 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:14:32.117856    4521 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-54dql"]
Oct 30 21:14:32.129986 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:14:32.129957    4521 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-sq68x"]
Oct 30 21:14:32.135162 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:14:32.135139    4521 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-sq68x"]
Oct 30 21:14:32.792776 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:14:32.792745    4521 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16e33702-94a7-4c37-b430-1ac4f002832b" path="/var/lib/kubelet/pods/16e33702-94a7-4c37-b430-1ac4f002832b/volumes"
Oct 30 21:14:32.793153 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:14:32.793140    4521 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be342bb0-ebf7-4bb3-a929-5abfc455531e" path="/var/lib/kubelet/pods/be342bb0-ebf7-4bb3-a929-5abfc455531e/volumes"
Oct 30 21:14:35.703901 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:14:35.703863    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4kxb"]
Oct 30 21:14:46.429121 ip-10-0-131-125 kubenswrapper[4521]: E1030 21:14:46.429080    4521 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lkn2k" containerName="extractor"
Oct 30 21:14:48.749040 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:14:48.748997    4521 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-49a6a4/simulator-single-main-49a6a4-kserve-85494f98d7-4kkmg"]
Oct 30 21:14:48.749579 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:14:48.749558    4521 cpu_manager.go:414] "containerMap: removing container" podUID="be342bb0-ebf7-4bb3-a929-5abfc455531e" containerName="istio-proxy"
Oct 30 21:14:48.749628 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:14:48.749583    4521 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be342bb0-ebf7-4bb3-a929-5abfc455531e" containerName="istio-proxy"
Oct 30 21:14:48.749628 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:14:48.749598    4521 cpu_manager.go:414] "containerMap: removing container" podUID="775d6d2d-e3eb-45b6-aeae-028f211a1f74" containerName="revoke-keys"
Oct 30 21:14:48.749628 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:14:48.749607    4521 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="775d6d2d-e3eb-45b6-aeae-028f211a1f74" containerName="revoke-keys"
Oct 30 21:14:48.749628 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:14:48.749623    4521 cpu_manager.go:414] "containerMap: removing container" podUID="16e33702-94a7-4c37-b430-1ac4f002832b" containerName="istio-proxy"
Oct 30 21:14:48.749755 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:14:48.749631    4521 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16e33702-94a7-4c37-b430-1ac4f002832b" containerName="istio-proxy"
Oct 30 21:14:48.749755 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:14:48.749735    4521 memory_manager.go:365] "RemoveStaleState removing state" podUID="be342bb0-ebf7-4bb3-a929-5abfc455531e" containerName="istio-proxy"
Oct 30 21:14:48.749755 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:14:48.749747    4521 memory_manager.go:365] "RemoveStaleState removing state" podUID="775d6d2d-e3eb-45b6-aeae-028f211a1f74" containerName="revoke-keys"
Oct 30 21:14:48.749848 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:14:48.749756    4521 memory_manager.go:365] "RemoveStaleState removing state" podUID="16e33702-94a7-4c37-b430-1ac4f002832b" containerName="istio-proxy"
Oct 30 21:14:48.753058 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:14:48.753035    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-49a6a4/simulator-single-main-49a6a4-kserve-85494f98d7-4kkmg"
Oct 30 21:14:48.759400 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:14:48.759377    4521 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-49a6a4\"/\"kube-root-ca.crt\""
Oct 30 21:14:48.759555 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:14:48.759516    4521 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-49a6a4\"/\"default-dockercfg-6z42h\""
Oct 30 21:14:48.759633 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:14:48.759570    4521 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-49a6a4\"/\"simulator-single-main-49a6a4-kserve-self-signed-certs\""
Oct 30 21:14:48.759633 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:14:48.759610    4521 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-49a6a4\"/\"openshift-service-ca.crt\""
Oct 30 21:14:48.762548 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:14:48.762514    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-49a6a4/simulator-single-main-49a6a4-kserve-85494f98d7-4kkmg"]
Oct 30 21:14:48.833523 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:14:48.833464    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4h8j4\" (UniqueName: \"kubernetes.io/projected/ed978d82-daa4-4d8a-90a7-a13e95243a00-kube-api-access-4h8j4\") pod \"simulator-single-main-49a6a4-kserve-85494f98d7-4kkmg\" (UID: \"ed978d82-daa4-4d8a-90a7-a13e95243a00\") " pod="e2e-models-e2e-worker-single-main-49a6a4/simulator-single-main-49a6a4-kserve-85494f98d7-4kkmg"
Oct 30 21:14:48.833523 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:14:48.833508    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed978d82-daa4-4d8a-90a7-a13e95243a00-home\") pod \"simulator-single-main-49a6a4-kserve-85494f98d7-4kkmg\" (UID: \"ed978d82-daa4-4d8a-90a7-a13e95243a00\") " pod="e2e-models-e2e-worker-single-main-49a6a4/simulator-single-main-49a6a4-kserve-85494f98d7-4kkmg"
Oct 30 21:14:48.833825 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:14:48.833536    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed978d82-daa4-4d8a-90a7-a13e95243a00-model-cache\") pod \"simulator-single-main-49a6a4-kserve-85494f98d7-4kkmg\" (UID: \"ed978d82-daa4-4d8a-90a7-a13e95243a00\") " pod="e2e-models-e2e-worker-single-main-49a6a4/simulator-single-main-49a6a4-kserve-85494f98d7-4kkmg"
Oct 30 21:14:48.833825 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:14:48.833595    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed978d82-daa4-4d8a-90a7-a13e95243a00-tls-certs\") pod \"simulator-single-main-49a6a4-kserve-85494f98d7-4kkmg\" (UID: \"ed978d82-daa4-4d8a-90a7-a13e95243a00\") " pod="e2e-models-e2e-worker-single-main-49a6a4/simulator-single-main-49a6a4-kserve-85494f98d7-4kkmg"
Oct 30 21:14:48.833825 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:14:48.833642    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed978d82-daa4-4d8a-90a7-a13e95243a00-tmp-dir\") pod \"simulator-single-main-49a6a4-kserve-85494f98d7-4kkmg\" (UID: \"ed978d82-daa4-4d8a-90a7-a13e95243a00\") " pod="e2e-models-e2e-worker-single-main-49a6a4/simulator-single-main-49a6a4-kserve-85494f98d7-4kkmg"
Oct 30 21:14:48.833825 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:14:48.833668    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed978d82-daa4-4d8a-90a7-a13e95243a00-dshm\") pod \"simulator-single-main-49a6a4-kserve-85494f98d7-4kkmg\" (UID: \"ed978d82-daa4-4d8a-90a7-a13e95243a00\") " pod="e2e-models-e2e-worker-single-main-49a6a4/simulator-single-main-49a6a4-kserve-85494f98d7-4kkmg"
Oct 30 21:14:48.934786 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:14:48.934744    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed978d82-daa4-4d8a-90a7-a13e95243a00-tmp-dir\") pod \"simulator-single-main-49a6a4-kserve-85494f98d7-4kkmg\" (UID: \"ed978d82-daa4-4d8a-90a7-a13e95243a00\") " pod="e2e-models-e2e-worker-single-main-49a6a4/simulator-single-main-49a6a4-kserve-85494f98d7-4kkmg"
Oct 30 21:14:48.934786 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:14:48.934786    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed978d82-daa4-4d8a-90a7-a13e95243a00-dshm\") pod \"simulator-single-main-49a6a4-kserve-85494f98d7-4kkmg\" (UID: \"ed978d82-daa4-4d8a-90a7-a13e95243a00\") " pod="e2e-models-e2e-worker-single-main-49a6a4/simulator-single-main-49a6a4-kserve-85494f98d7-4kkmg"
Oct 30 21:14:48.935071 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:14:48.934848    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4h8j4\" (UniqueName: \"kubernetes.io/projected/ed978d82-daa4-4d8a-90a7-a13e95243a00-kube-api-access-4h8j4\") pod \"simulator-single-main-49a6a4-kserve-85494f98d7-4kkmg\" (UID: \"ed978d82-daa4-4d8a-90a7-a13e95243a00\") " pod="e2e-models-e2e-worker-single-main-49a6a4/simulator-single-main-49a6a4-kserve-85494f98d7-4kkmg"
Oct 30 21:14:48.935071 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:14:48.934875    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed978d82-daa4-4d8a-90a7-a13e95243a00-home\") pod \"simulator-single-main-49a6a4-kserve-85494f98d7-4kkmg\" (UID: \"ed978d82-daa4-4d8a-90a7-a13e95243a00\") " pod="e2e-models-e2e-worker-single-main-49a6a4/simulator-single-main-49a6a4-kserve-85494f98d7-4kkmg"
Oct 30 21:14:48.935071 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:14:48.935028    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed978d82-daa4-4d8a-90a7-a13e95243a00-model-cache\") pod \"simulator-single-main-49a6a4-kserve-85494f98d7-4kkmg\" (UID: \"ed978d82-daa4-4d8a-90a7-a13e95243a00\") " pod="e2e-models-e2e-worker-single-main-49a6a4/simulator-single-main-49a6a4-kserve-85494f98d7-4kkmg"
Oct 30 21:14:48.935240 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:14:48.935101    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed978d82-daa4-4d8a-90a7-a13e95243a00-tls-certs\") pod \"simulator-single-main-49a6a4-kserve-85494f98d7-4kkmg\" (UID: \"ed978d82-daa4-4d8a-90a7-a13e95243a00\") " pod="e2e-models-e2e-worker-single-main-49a6a4/simulator-single-main-49a6a4-kserve-85494f98d7-4kkmg"
Oct 30 21:14:48.935298 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:14:48.935239    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed978d82-daa4-4d8a-90a7-a13e95243a00-tmp-dir\") pod \"simulator-single-main-49a6a4-kserve-85494f98d7-4kkmg\" (UID: \"ed978d82-daa4-4d8a-90a7-a13e95243a00\") " pod="e2e-models-e2e-worker-single-main-49a6a4/simulator-single-main-49a6a4-kserve-85494f98d7-4kkmg"
Oct 30 21:14:48.935541 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:14:48.935516    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed978d82-daa4-4d8a-90a7-a13e95243a00-model-cache\") pod \"simulator-single-main-49a6a4-kserve-85494f98d7-4kkmg\" (UID: \"ed978d82-daa4-4d8a-90a7-a13e95243a00\") " pod="e2e-models-e2e-worker-single-main-49a6a4/simulator-single-main-49a6a4-kserve-85494f98d7-4kkmg"
Oct 30 21:14:48.935605 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:14:48.935554    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed978d82-daa4-4d8a-90a7-a13e95243a00-home\") pod \"simulator-single-main-49a6a4-kserve-85494f98d7-4kkmg\" (UID: \"ed978d82-daa4-4d8a-90a7-a13e95243a00\") " pod="e2e-models-e2e-worker-single-main-49a6a4/simulator-single-main-49a6a4-kserve-85494f98d7-4kkmg"
Oct 30 21:14:48.937562 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:14:48.937538    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed978d82-daa4-4d8a-90a7-a13e95243a00-dshm\") pod \"simulator-single-main-49a6a4-kserve-85494f98d7-4kkmg\" (UID: \"ed978d82-daa4-4d8a-90a7-a13e95243a00\") " pod="e2e-models-e2e-worker-single-main-49a6a4/simulator-single-main-49a6a4-kserve-85494f98d7-4kkmg"
Oct 30 21:14:48.937677 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:14:48.937600    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed978d82-daa4-4d8a-90a7-a13e95243a00-tls-certs\") pod \"simulator-single-main-49a6a4-kserve-85494f98d7-4kkmg\" (UID: \"ed978d82-daa4-4d8a-90a7-a13e95243a00\") " pod="e2e-models-e2e-worker-single-main-49a6a4/simulator-single-main-49a6a4-kserve-85494f98d7-4kkmg"
Oct 30 21:14:48.947222 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:14:48.947196    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4h8j4\" (UniqueName: \"kubernetes.io/projected/ed978d82-daa4-4d8a-90a7-a13e95243a00-kube-api-access-4h8j4\") pod \"simulator-single-main-49a6a4-kserve-85494f98d7-4kkmg\" (UID: \"ed978d82-daa4-4d8a-90a7-a13e95243a00\") " pod="e2e-models-e2e-worker-single-main-49a6a4/simulator-single-main-49a6a4-kserve-85494f98d7-4kkmg"
Oct 30 21:14:49.068412 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:14:49.068305    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-49a6a4/simulator-single-main-49a6a4-kserve-85494f98d7-4kkmg"
Oct 30 21:14:49.354788 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:14:49.354759    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-49a6a4/simulator-single-main-49a6a4-kserve-85494f98d7-4kkmg"]
Oct 30 21:14:49.356568 ip-10-0-131-125 kubenswrapper[4521]: W1030 21:14:49.356542    4521 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poded978d82_daa4_4d8a_90a7_a13e95243a00.slice/crio-0576e92ac24fac848cc724bdac009e0f499c7adc889520cf398abf59e8a0304d WatchSource:0}: Error finding container 0576e92ac24fac848cc724bdac009e0f499c7adc889520cf398abf59e8a0304d: Status 404 returned error can't find the container with id 0576e92ac24fac848cc724bdac009e0f499c7adc889520cf398abf59e8a0304d
Oct 30 21:14:50.187599 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:14:50.187559    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-49a6a4/simulator-single-main-49a6a4-kserve-85494f98d7-4kkmg" event={"ID":"ed978d82-daa4-4d8a-90a7-a13e95243a00","Type":"ContainerStarted","Data":"3c6c4609656c3d7915a7a4203a89f3f685c77a72bb666417af6e577333e8079b"}
Oct 30 21:14:50.188080 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:14:50.187612    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-49a6a4/simulator-single-main-49a6a4-kserve-85494f98d7-4kkmg" event={"ID":"ed978d82-daa4-4d8a-90a7-a13e95243a00","Type":"ContainerStarted","Data":"0576e92ac24fac848cc724bdac009e0f499c7adc889520cf398abf59e8a0304d"}
Oct 30 21:14:50.223704 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:14:50.223640    4521 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-49a6a4/simulator-single-main-49a6a4-kserve-85494f98d7-4kkmg" podStartSLOduration=2.223622114 podStartE2EDuration="2.223622114s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:14:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:14:50.222945132 +0000 UTC m=+2157.568566370" watchObservedRunningTime="2026-10-30 21:14:50.223622114 +0000 UTC m=+2157.569243352"
Oct 30 21:14:50.424309 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:14:50.424269    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4kxb"]
Oct 30 21:14:51.069303 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:14:51.069265    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-49a6a4/simulator-single-main-49a6a4-kserve-85494f98d7-4kkmg"
Oct 30 21:14:54.665804 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:14:54.665772    4521 scope.go:122] "RemoveContainer" containerID="739064a2d8a063d5dc9424bfc3975a6e3a8c668e4ae149dec50cbc2b43e66b8c"
Oct 30 21:14:54.674525 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:14:54.674504    4521 scope.go:122] "RemoveContainer" containerID="2fb97a194a31e2ebdce192551e07a627ec78921de418b315c4ebd2603af9d7ae"
Oct 30 21:14:59.068517 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:14:59.068485    4521 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-49a6a4/simulator-single-main-49a6a4-kserve-85494f98d7-4kkmg"
Oct 30 21:14:59.080983 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:14:59.080960    4521 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-49a6a4/simulator-single-main-49a6a4-kserve-85494f98d7-4kkmg"
Oct 30 21:14:59.243861 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:14:59.243829    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-49a6a4/simulator-single-main-49a6a4-kserve-85494f98d7-4kkmg"
Oct 30 21:15:00.140188 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:15:00.140139    4521 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889915-7tsrn"]
Oct 30 21:15:00.145601 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:15:00.145577    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-7tsrn"
Oct 30 21:15:00.153110 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:15:00.153087    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889915-7tsrn"]
Oct 30 21:15:00.153419 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:15:00.153385    4521 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:15:00.153533 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:15:00.153435    4521 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:15:00.153533 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:15:00.153460    4521 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c45pw\""
Oct 30 21:15:00.237483 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:15:00.237449    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bd76a0a-a068-41a3-a7a7-10e473622c7d-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889915-7tsrn\" (UID: \"4bd76a0a-a068-41a3-a7a7-10e473622c7d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-7tsrn"
Oct 30 21:15:00.237690 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:15:00.237496    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgp4v\" (UniqueName: \"kubernetes.io/projected/4bd76a0a-a068-41a3-a7a7-10e473622c7d-kube-api-access-fgp4v\") pod \"maas-api-key-cleanup-29889915-7tsrn\" (UID: \"4bd76a0a-a068-41a3-a7a7-10e473622c7d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-7tsrn"
Oct 30 21:15:00.338228 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:15:00.338189    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bd76a0a-a068-41a3-a7a7-10e473622c7d-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889915-7tsrn\" (UID: \"4bd76a0a-a068-41a3-a7a7-10e473622c7d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-7tsrn"
Oct 30 21:15:00.338524 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:15:00.338500    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgp4v\" (UniqueName: \"kubernetes.io/projected/4bd76a0a-a068-41a3-a7a7-10e473622c7d-kube-api-access-fgp4v\") pod \"maas-api-key-cleanup-29889915-7tsrn\" (UID: \"4bd76a0a-a068-41a3-a7a7-10e473622c7d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-7tsrn"
Oct 30 21:15:00.338947 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:15:00.338926    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bd76a0a-a068-41a3-a7a7-10e473622c7d-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889915-7tsrn\" (UID: \"4bd76a0a-a068-41a3-a7a7-10e473622c7d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-7tsrn"
Oct 30 21:15:00.353905 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:15:00.353879    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgp4v\" (UniqueName: \"kubernetes.io/projected/4bd76a0a-a068-41a3-a7a7-10e473622c7d-kube-api-access-fgp4v\") pod \"maas-api-key-cleanup-29889915-7tsrn\" (UID: \"4bd76a0a-a068-41a3-a7a7-10e473622c7d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-7tsrn"
Oct 30 21:15:00.461623 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:15:00.461587    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-7tsrn"
Oct 30 21:15:00.735084 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:15:00.735052    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889915-7tsrn"]
Oct 30 21:15:00.736438 ip-10-0-131-125 kubenswrapper[4521]: W1030 21:15:00.736414    4521 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bd76a0a_a068_41a3_a7a7_10e473622c7d.slice/crio-e6cfd1eb04a99a7fd136464a6757583ecd83d9f2f7c2ba5e390bc5b5461703c6 WatchSource:0}: Error finding container e6cfd1eb04a99a7fd136464a6757583ecd83d9f2f7c2ba5e390bc5b5461703c6: Status 404 returned error can't find the container with id e6cfd1eb04a99a7fd136464a6757583ecd83d9f2f7c2ba5e390bc5b5461703c6
Oct 30 21:15:01.244861 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:15:01.244830    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-7tsrn" event={"ID":"4bd76a0a-a068-41a3-a7a7-10e473622c7d","Type":"ContainerStarted","Data":"fe938351d2f13d7c088bc4e0a665e5354d1ad6d1ed58b4e6f8cd41e4d5e0c8d6"}
Oct 30 21:15:01.245277 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:15:01.244879    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-7tsrn" event={"ID":"4bd76a0a-a068-41a3-a7a7-10e473622c7d","Type":"ContainerStarted","Data":"e6cfd1eb04a99a7fd136464a6757583ecd83d9f2f7c2ba5e390bc5b5461703c6"}
Oct 30 21:15:01.263648 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:15:01.263602    4521 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-7tsrn" podStartSLOduration=1.2635898239999999 podStartE2EDuration="1.263589824s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:15:01.262873317 +0000 UTC m=+2168.608494557" watchObservedRunningTime="2026-10-30 21:15:01.263589824 +0000 UTC m=+2168.609211061"
Oct 30 21:15:02.251077 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:15:02.251040    4521 generic.go:356] "Generic (PLEG): container finished" podID="4bd76a0a-a068-41a3-a7a7-10e473622c7d" containerID="fe938351d2f13d7c088bc4e0a665e5354d1ad6d1ed58b4e6f8cd41e4d5e0c8d6" exitCode=60
Oct 30 21:15:02.251510 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:15:02.251127    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-7tsrn" event={"ID":"4bd76a0a-a068-41a3-a7a7-10e473622c7d","Type":"ContainerDied","Data":"fe938351d2f13d7c088bc4e0a665e5354d1ad6d1ed58b4e6f8cd41e4d5e0c8d6"}
Oct 30 21:15:02.251510 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:15:02.251379    4521 scope.go:122] "RemoveContainer" containerID="fe938351d2f13d7c088bc4e0a665e5354d1ad6d1ed58b4e6f8cd41e4d5e0c8d6"
Oct 30 21:15:03.257651 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:15:03.257619    4521 generic.go:356] "Generic (PLEG): container finished" podID="4bd76a0a-a068-41a3-a7a7-10e473622c7d" containerID="79eb1a97cceff1c1fc1a59cf99a148539041ae369a5eae31821070a346a8e15c" exitCode=60
Oct 30 21:15:03.258143 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:15:03.257687    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-7tsrn" event={"ID":"4bd76a0a-a068-41a3-a7a7-10e473622c7d","Type":"ContainerDied","Data":"79eb1a97cceff1c1fc1a59cf99a148539041ae369a5eae31821070a346a8e15c"}
Oct 30 21:15:03.258143 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:15:03.257763    4521 scope.go:122] "RemoveContainer" containerID="fe938351d2f13d7c088bc4e0a665e5354d1ad6d1ed58b4e6f8cd41e4d5e0c8d6"
Oct 30 21:15:03.258143 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:15:03.258021    4521 scope.go:122] "RemoveContainer" containerID="79eb1a97cceff1c1fc1a59cf99a148539041ae369a5eae31821070a346a8e15c"
Oct 30 21:15:03.258273 ip-10-0-131-125 kubenswrapper[4521]: E1030 21:15:03.258256    4521 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889915-7tsrn_odh-ai-gateway-infra(4bd76a0a-a068-41a3-a7a7-10e473622c7d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-7tsrn" podUID="4bd76a0a-a068-41a3-a7a7-10e473622c7d"
Oct 30 21:15:04.264259 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:15:04.264229    4521 scope.go:122] "RemoveContainer" containerID="79eb1a97cceff1c1fc1a59cf99a148539041ae369a5eae31821070a346a8e15c"
Oct 30 21:15:04.264661 ip-10-0-131-125 kubenswrapper[4521]: E1030 21:15:04.264466    4521 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889915-7tsrn_odh-ai-gateway-infra(4bd76a0a-a068-41a3-a7a7-10e473622c7d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-7tsrn" podUID="4bd76a0a-a068-41a3-a7a7-10e473622c7d"
Oct 30 21:15:07.412824 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:15:07.412787    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4kxb"]
Oct 30 21:15:14.785749 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:15:14.785718    4521 scope.go:122] "RemoveContainer" containerID="79eb1a97cceff1c1fc1a59cf99a148539041ae369a5eae31821070a346a8e15c"
Oct 30 21:15:15.321189 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:15:15.321147    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-7tsrn" event={"ID":"4bd76a0a-a068-41a3-a7a7-10e473622c7d","Type":"ContainerStarted","Data":"4d7980bfbbe2ec79be3354888ba4c8a680d86c2d025e33d7ef07d1d163d24d36"}
Oct 30 21:15:16.327964 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:15:16.327929    4521 generic.go:356] "Generic (PLEG): container finished" podID="4bd76a0a-a068-41a3-a7a7-10e473622c7d" containerID="4d7980bfbbe2ec79be3354888ba4c8a680d86c2d025e33d7ef07d1d163d24d36" exitCode=60
Oct 30 21:15:16.328408 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:15:16.327998    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-7tsrn" event={"ID":"4bd76a0a-a068-41a3-a7a7-10e473622c7d","Type":"ContainerDied","Data":"4d7980bfbbe2ec79be3354888ba4c8a680d86c2d025e33d7ef07d1d163d24d36"}
Oct 30 21:15:16.328408 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:15:16.328080    4521 scope.go:122] "RemoveContainer" containerID="79eb1a97cceff1c1fc1a59cf99a148539041ae369a5eae31821070a346a8e15c"
Oct 30 21:15:16.328508 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:15:16.328452    4521 scope.go:122] "RemoveContainer" containerID="4d7980bfbbe2ec79be3354888ba4c8a680d86c2d025e33d7ef07d1d163d24d36"
Oct 30 21:15:16.328726 ip-10-0-131-125 kubenswrapper[4521]: E1030 21:15:16.328702    4521 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889915-7tsrn_odh-ai-gateway-infra(4bd76a0a-a068-41a3-a7a7-10e473622c7d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-7tsrn" podUID="4bd76a0a-a068-41a3-a7a7-10e473622c7d"
Oct 30 21:15:16.357457 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:15:16.357429    4521 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889915-7tsrn"]
Oct 30 21:15:16.431061 ip-10-0-131-125 kubenswrapper[4521]: E1030 21:15:16.430945    4521 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lkn2k" containerName="extractor"
Oct 30 21:15:17.464748 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:15:17.464721    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-7tsrn"
Oct 30 21:15:17.589547 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:15:17.589466    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4bd76a0a-a068-41a3-a7a7-10e473622c7d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bd76a0a-a068-41a3-a7a7-10e473622c7d-maas-api-service-ca\") pod \"4bd76a0a-a068-41a3-a7a7-10e473622c7d\" (UID: \"4bd76a0a-a068-41a3-a7a7-10e473622c7d\") "
Oct 30 21:15:17.589547 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:15:17.589510    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bd76a0a-a068-41a3-a7a7-10e473622c7d-kube-api-access-fgp4v\" (UniqueName: \"kubernetes.io/projected/4bd76a0a-a068-41a3-a7a7-10e473622c7d-kube-api-access-fgp4v\") pod \"4bd76a0a-a068-41a3-a7a7-10e473622c7d\" (UID: \"4bd76a0a-a068-41a3-a7a7-10e473622c7d\") "
Oct 30 21:15:17.589828 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:15:17.589804    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bd76a0a-a068-41a3-a7a7-10e473622c7d-maas-api-service-ca" pod "4bd76a0a-a068-41a3-a7a7-10e473622c7d" (UID: "4bd76a0a-a068-41a3-a7a7-10e473622c7d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:15:17.591612 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:15:17.591592    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bd76a0a-a068-41a3-a7a7-10e473622c7d-kube-api-access-fgp4v" pod "4bd76a0a-a068-41a3-a7a7-10e473622c7d" (UID: "4bd76a0a-a068-41a3-a7a7-10e473622c7d"). InnerVolumeSpecName "kube-api-access-fgp4v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:15:17.690276 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:15:17.690240    4521 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bd76a0a-a068-41a3-a7a7-10e473622c7d-maas-api-service-ca\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:15:17.690276 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:15:17.690269    4521 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgp4v\" (UniqueName: \"kubernetes.io/projected/4bd76a0a-a068-41a3-a7a7-10e473622c7d-kube-api-access-fgp4v\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:15:18.341085 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:15:18.341054    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-7tsrn"
Oct 30 21:15:18.341275 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:15:18.341104    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-7tsrn" event={"ID":"4bd76a0a-a068-41a3-a7a7-10e473622c7d","Type":"ContainerDied","Data":"e6cfd1eb04a99a7fd136464a6757583ecd83d9f2f7c2ba5e390bc5b5461703c6"}
Oct 30 21:15:18.341275 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:15:18.341169    4521 scope.go:122] "RemoveContainer" containerID="4d7980bfbbe2ec79be3354888ba4c8a680d86c2d025e33d7ef07d1d163d24d36"
Oct 30 21:15:18.365504 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:15:18.365475    4521 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889915-7tsrn"]
Oct 30 21:15:18.371975 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:15:18.371949    4521 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889915-7tsrn"]
Oct 30 21:15:18.790257 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:15:18.790226    4521 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4bd76a0a-a068-41a3-a7a7-10e473622c7d" path="/var/lib/kubelet/pods/4bd76a0a-a068-41a3-a7a7-10e473622c7d/volumes"
Oct 30 21:15:18.850272 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:15:18.850236    4521 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7396a035-p69ls"]
Oct 30 21:15:18.850668 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:15:18.850651    4521 cpu_manager.go:414] "containerMap: removing container" podUID="4bd76a0a-a068-41a3-a7a7-10e473622c7d" containerName="cleanup"
Oct 30 21:15:18.850731 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:15:18.850669    4521 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bd76a0a-a068-41a3-a7a7-10e473622c7d" containerName="cleanup"
Oct 30 21:15:18.850731 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:15:18.850680    4521 cpu_manager.go:414] "containerMap: removing container" podUID="4bd76a0a-a068-41a3-a7a7-10e473622c7d" containerName="cleanup"
Oct 30 21:15:18.850731 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:15:18.850685    4521 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bd76a0a-a068-41a3-a7a7-10e473622c7d" containerName="cleanup"
Oct 30 21:15:18.850825 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:15:18.850751    4521 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bd76a0a-a068-41a3-a7a7-10e473622c7d" containerName="cleanup"
Oct 30 21:15:18.850825 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:15:18.850759    4521 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bd76a0a-a068-41a3-a7a7-10e473622c7d" containerName="cleanup"
Oct 30 21:15:18.854850 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:15:18.854833    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7396a035-p69ls"
Oct 30 21:15:18.861146 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:15:18.861117    4521 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:15:18.861264 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:15:18.861205    4521 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:15:18.861264 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:15:18.861209    4521 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c45pw\""
Oct 30 21:15:18.861858 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:15:18.861836    4521 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:15:18.864256 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:15:18.864228    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7396a035-p69ls"]
Oct 30 21:15:19.002409 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:15:19.002358    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lftg\" (UniqueName: \"kubernetes.io/projected/e261a5da-939e-45a8-a988-f520f446490b-kube-api-access-6lftg\") pod \"maas-api-revoke-sub-7396a035-p69ls\" (UID: \"e261a5da-939e-45a8-a988-f520f446490b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7396a035-p69ls"
Oct 30 21:15:19.002598 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:15:19.002472    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e261a5da-939e-45a8-a988-f520f446490b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7396a035-p69ls\" (UID: \"e261a5da-939e-45a8-a988-f520f446490b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7396a035-p69ls"
Oct 30 21:15:19.002598 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:15:19.002528    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e261a5da-939e-45a8-a988-f520f446490b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7396a035-p69ls\" (UID: \"e261a5da-939e-45a8-a988-f520f446490b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7396a035-p69ls"
Oct 30 21:15:19.103374 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:15:19.103269    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e261a5da-939e-45a8-a988-f520f446490b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7396a035-p69ls\" (UID: \"e261a5da-939e-45a8-a988-f520f446490b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7396a035-p69ls"
Oct 30 21:15:19.103374 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:15:19.103358    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e261a5da-939e-45a8-a988-f520f446490b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7396a035-p69ls\" (UID: \"e261a5da-939e-45a8-a988-f520f446490b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7396a035-p69ls"
Oct 30 21:15:19.103619 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:15:19.103437    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lftg\" (UniqueName: \"kubernetes.io/projected/e261a5da-939e-45a8-a988-f520f446490b-kube-api-access-6lftg\") pod \"maas-api-revoke-sub-7396a035-p69ls\" (UID: \"e261a5da-939e-45a8-a988-f520f446490b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7396a035-p69ls"
Oct 30 21:15:19.103922 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:15:19.103901    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e261a5da-939e-45a8-a988-f520f446490b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7396a035-p69ls\" (UID: \"e261a5da-939e-45a8-a988-f520f446490b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7396a035-p69ls"
Oct 30 21:15:19.105747 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:15:19.105723    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e261a5da-939e-45a8-a988-f520f446490b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7396a035-p69ls\" (UID: \"e261a5da-939e-45a8-a988-f520f446490b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7396a035-p69ls"
Oct 30 21:15:19.115404 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:15:19.115376    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lftg\" (UniqueName: \"kubernetes.io/projected/e261a5da-939e-45a8-a988-f520f446490b-kube-api-access-6lftg\") pod \"maas-api-revoke-sub-7396a035-p69ls\" (UID: \"e261a5da-939e-45a8-a988-f520f446490b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7396a035-p69ls"
Oct 30 21:15:19.168551 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:15:19.168524    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7396a035-p69ls"
Oct 30 21:15:19.443466 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:15:19.443441    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7396a035-p69ls"]
Oct 30 21:15:19.445811 ip-10-0-131-125 kubenswrapper[4521]: W1030 21:15:19.445773    4521 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode261a5da_939e_45a8_a988_f520f446490b.slice/crio-9897af2fe9dbd650c5b04e848f70ef1554ffd4c9ee13ae487de0dbc3d4b1e4cf WatchSource:0}: Error finding container 9897af2fe9dbd650c5b04e848f70ef1554ffd4c9ee13ae487de0dbc3d4b1e4cf: Status 404 returned error can't find the container with id 9897af2fe9dbd650c5b04e848f70ef1554ffd4c9ee13ae487de0dbc3d4b1e4cf
Oct 30 21:15:20.357723 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:15:20.357687    4521 generic.go:356] "Generic (PLEG): container finished" podID="e261a5da-939e-45a8-a988-f520f446490b" containerID="f80614565e3c6f468f3662eb052d2d1d3f61884265918f1431df49e3be632bc0" exitCode=0
Oct 30 21:15:20.358111 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:15:20.357727    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7396a035-p69ls" event={"ID":"e261a5da-939e-45a8-a988-f520f446490b","Type":"ContainerDied","Data":"f80614565e3c6f468f3662eb052d2d1d3f61884265918f1431df49e3be632bc0"}
Oct 30 21:15:20.358111 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:15:20.357778    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7396a035-p69ls" event={"ID":"e261a5da-939e-45a8-a988-f520f446490b","Type":"ContainerStarted","Data":"9897af2fe9dbd650c5b04e848f70ef1554ffd4c9ee13ae487de0dbc3d4b1e4cf"}
Oct 30 21:15:21.489075 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:15:21.489046    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7396a035-p69ls"
Oct 30 21:15:21.522911 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:15:21.522884    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e261a5da-939e-45a8-a988-f520f446490b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e261a5da-939e-45a8-a988-f520f446490b-maas-api-openshift-ca\") pod \"e261a5da-939e-45a8-a988-f520f446490b\" (UID: \"e261a5da-939e-45a8-a988-f520f446490b\") "
Oct 30 21:15:21.523072 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:15:21.522957    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e261a5da-939e-45a8-a988-f520f446490b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e261a5da-939e-45a8-a988-f520f446490b-maas-api-xks-ca\") pod \"e261a5da-939e-45a8-a988-f520f446490b\" (UID: \"e261a5da-939e-45a8-a988-f520f446490b\") "
Oct 30 21:15:21.523245 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:15:21.523224    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e261a5da-939e-45a8-a988-f520f446490b-maas-api-openshift-ca" pod "e261a5da-939e-45a8-a988-f520f446490b" (UID: "e261a5da-939e-45a8-a988-f520f446490b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:15:21.525161 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:15:21.525129    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e261a5da-939e-45a8-a988-f520f446490b-maas-api-xks-ca" pod "e261a5da-939e-45a8-a988-f520f446490b" (UID: "e261a5da-939e-45a8-a988-f520f446490b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:15:21.624093 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:15:21.624014    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e261a5da-939e-45a8-a988-f520f446490b-kube-api-access-6lftg\" (UniqueName: \"kubernetes.io/projected/e261a5da-939e-45a8-a988-f520f446490b-kube-api-access-6lftg\") pod \"e261a5da-939e-45a8-a988-f520f446490b\" (UID: \"e261a5da-939e-45a8-a988-f520f446490b\") "
Oct 30 21:15:21.624253 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:15:21.624190    4521 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e261a5da-939e-45a8-a988-f520f446490b-maas-api-xks-ca\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:15:21.624253 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:15:21.624205    4521 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e261a5da-939e-45a8-a988-f520f446490b-maas-api-openshift-ca\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:15:21.626058 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:15:21.626031    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e261a5da-939e-45a8-a988-f520f446490b-kube-api-access-6lftg" pod "e261a5da-939e-45a8-a988-f520f446490b" (UID: "e261a5da-939e-45a8-a988-f520f446490b"). InnerVolumeSpecName "kube-api-access-6lftg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:15:21.724777 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:15:21.724742    4521 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6lftg\" (UniqueName: \"kubernetes.io/projected/e261a5da-939e-45a8-a988-f520f446490b-kube-api-access-6lftg\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:15:22.369765 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:15:22.369734    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7396a035-p69ls"
Oct 30 21:15:22.369942 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:15:22.369771    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7396a035-p69ls" event={"ID":"e261a5da-939e-45a8-a988-f520f446490b","Type":"ContainerDied","Data":"9897af2fe9dbd650c5b04e848f70ef1554ffd4c9ee13ae487de0dbc3d4b1e4cf"}
Oct 30 21:15:22.369942 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:15:22.369817    4521 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9897af2fe9dbd650c5b04e848f70ef1554ffd4c9ee13ae487de0dbc3d4b1e4cf"
Oct 30 21:15:29.918755 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:15:29.918704    4521 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-49a6a4-q252h"]
Oct 30 21:15:29.919279 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:15:29.919067    4521 cpu_manager.go:414] "containerMap: removing container" podUID="4bd76a0a-a068-41a3-a7a7-10e473622c7d" containerName="cleanup"
Oct 30 21:15:29.919279 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:15:29.919078    4521 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bd76a0a-a068-41a3-a7a7-10e473622c7d" containerName="cleanup"
Oct 30 21:15:29.919279 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:15:29.919107    4521 cpu_manager.go:414] "containerMap: removing container" podUID="e261a5da-939e-45a8-a988-f520f446490b" containerName="revoke-keys"
Oct 30 21:15:29.919279 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:15:29.919112    4521 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e261a5da-939e-45a8-a988-f520f446490b" containerName="revoke-keys"
Oct 30 21:15:29.919279 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:15:29.919178    4521 memory_manager.go:365] "RemoveStaleState removing state" podUID="e261a5da-939e-45a8-a988-f520f446490b" containerName="revoke-keys"
Oct 30 21:15:29.919279 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:15:29.919189    4521 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bd76a0a-a068-41a3-a7a7-10e473622c7d" containerName="cleanup"
Oct 30 21:15:29.922262 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:15:29.922246    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-49a6a4-q252h"
Oct 30 21:15:29.933504 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:15:29.933447    4521 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:15:29.933504 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:15:29.933457    4521 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:15:29.933710 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:15:29.933508    4521 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:15:29.933710 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:15:29.933546    4521 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c45pw\""
Oct 30 21:15:29.933710 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:15:29.933681    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-49a6a4-q252h"]
Oct 30 21:15:29.995237 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:15:29.995199    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kn6v9\" (UniqueName: \"kubernetes.io/projected/6843885e-1772-4f9e-893a-638a2bb9a821-kube-api-access-kn6v9\") pod \"maas-api-revoke-keys-e2e-worker-single-main-49a6a4-q252h\" (UID: \"6843885e-1772-4f9e-893a-638a2bb9a821\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-49a6a4-q252h"
Oct 30 21:15:29.995472 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:15:29.995264    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6843885e-1772-4f9e-893a-638a2bb9a821-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-49a6a4-q252h\" (UID: \"6843885e-1772-4f9e-893a-638a2bb9a821\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-49a6a4-q252h"
Oct 30 21:15:29.995472 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:15:29.995409    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6843885e-1772-4f9e-893a-638a2bb9a821-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-49a6a4-q252h\" (UID: \"6843885e-1772-4f9e-893a-638a2bb9a821\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-49a6a4-q252h"
Oct 30 21:15:30.096948 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:15:30.096915    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kn6v9\" (UniqueName: \"kubernetes.io/projected/6843885e-1772-4f9e-893a-638a2bb9a821-kube-api-access-kn6v9\") pod \"maas-api-revoke-keys-e2e-worker-single-main-49a6a4-q252h\" (UID: \"6843885e-1772-4f9e-893a-638a2bb9a821\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-49a6a4-q252h"
Oct 30 21:15:30.097175 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:15:30.096983    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6843885e-1772-4f9e-893a-638a2bb9a821-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-49a6a4-q252h\" (UID: \"6843885e-1772-4f9e-893a-638a2bb9a821\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-49a6a4-q252h"
Oct 30 21:15:30.097175 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:15:30.097012    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6843885e-1772-4f9e-893a-638a2bb9a821-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-49a6a4-q252h\" (UID: \"6843885e-1772-4f9e-893a-638a2bb9a821\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-49a6a4-q252h"
Oct 30 21:15:30.097679 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:15:30.097657    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6843885e-1772-4f9e-893a-638a2bb9a821-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-49a6a4-q252h\" (UID: \"6843885e-1772-4f9e-893a-638a2bb9a821\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-49a6a4-q252h"
Oct 30 21:15:30.099366 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:15:30.099343    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6843885e-1772-4f9e-893a-638a2bb9a821-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-49a6a4-q252h\" (UID: \"6843885e-1772-4f9e-893a-638a2bb9a821\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-49a6a4-q252h"
Oct 30 21:15:30.109034 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:15:30.109008    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kn6v9\" (UniqueName: \"kubernetes.io/projected/6843885e-1772-4f9e-893a-638a2bb9a821-kube-api-access-kn6v9\") pod \"maas-api-revoke-keys-e2e-worker-single-main-49a6a4-q252h\" (UID: \"6843885e-1772-4f9e-893a-638a2bb9a821\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-49a6a4-q252h"
Oct 30 21:15:30.241992 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:15:30.241947    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-49a6a4-q252h"
Oct 30 21:15:30.521313 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:15:30.521285    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-49a6a4-q252h"]
Oct 30 21:15:30.524252 ip-10-0-131-125 kubenswrapper[4521]: W1030 21:15:30.524214    4521 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6843885e_1772_4f9e_893a_638a2bb9a821.slice/crio-82eaec1bae2ffae61f6072eef5606a4f223ec00e974a538eff245d6bb5484f19 WatchSource:0}: Error finding container 82eaec1bae2ffae61f6072eef5606a4f223ec00e974a538eff245d6bb5484f19: Status 404 returned error can't find the container with id 82eaec1bae2ffae61f6072eef5606a4f223ec00e974a538eff245d6bb5484f19
Oct 30 21:15:30.688766 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:15:30.688732    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4kxb"]
Oct 30 21:15:31.415519 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:15:31.415481    4521 generic.go:356] "Generic (PLEG): container finished" podID="6843885e-1772-4f9e-893a-638a2bb9a821" containerID="1fd89f488e23fe6f3aba55c1ec6110f11acffd994306c75398b8b4486cc71e83" exitCode=0
Oct 30 21:15:31.415952 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:15:31.415555    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-49a6a4-q252h" event={"ID":"6843885e-1772-4f9e-893a-638a2bb9a821","Type":"ContainerDied","Data":"1fd89f488e23fe6f3aba55c1ec6110f11acffd994306c75398b8b4486cc71e83"}
Oct 30 21:15:31.415952 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:15:31.415609    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-49a6a4-q252h" event={"ID":"6843885e-1772-4f9e-893a-638a2bb9a821","Type":"ContainerStarted","Data":"82eaec1bae2ffae61f6072eef5606a4f223ec00e974a538eff245d6bb5484f19"}
Oct 30 21:15:32.563411 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:15:32.563385    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-49a6a4-q252h"
Oct 30 21:15:32.619469 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:15:32.619439    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6843885e-1772-4f9e-893a-638a2bb9a821-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6843885e-1772-4f9e-893a-638a2bb9a821-maas-api-openshift-ca\") pod \"6843885e-1772-4f9e-893a-638a2bb9a821\" (UID: \"6843885e-1772-4f9e-893a-638a2bb9a821\") "
Oct 30 21:15:32.619650 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:15:32.619484    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6843885e-1772-4f9e-893a-638a2bb9a821-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6843885e-1772-4f9e-893a-638a2bb9a821-maas-api-xks-ca\") pod \"6843885e-1772-4f9e-893a-638a2bb9a821\" (UID: \"6843885e-1772-4f9e-893a-638a2bb9a821\") "
Oct 30 21:15:32.619650 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:15:32.619623    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6843885e-1772-4f9e-893a-638a2bb9a821-kube-api-access-kn6v9\" (UniqueName: \"kubernetes.io/projected/6843885e-1772-4f9e-893a-638a2bb9a821-kube-api-access-kn6v9\") pod \"6843885e-1772-4f9e-893a-638a2bb9a821\" (UID: \"6843885e-1772-4f9e-893a-638a2bb9a821\") "
Oct 30 21:15:32.619878 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:15:32.619856    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6843885e-1772-4f9e-893a-638a2bb9a821-maas-api-openshift-ca" pod "6843885e-1772-4f9e-893a-638a2bb9a821" (UID: "6843885e-1772-4f9e-893a-638a2bb9a821"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:15:32.619984 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:15:32.619967    4521 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6843885e-1772-4f9e-893a-638a2bb9a821-maas-api-openshift-ca\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:15:32.621618 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:15:32.621590    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6843885e-1772-4f9e-893a-638a2bb9a821-maas-api-xks-ca" pod "6843885e-1772-4f9e-893a-638a2bb9a821" (UID: "6843885e-1772-4f9e-893a-638a2bb9a821"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:15:32.621618 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:15:32.621612    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6843885e-1772-4f9e-893a-638a2bb9a821-kube-api-access-kn6v9" pod "6843885e-1772-4f9e-893a-638a2bb9a821" (UID: "6843885e-1772-4f9e-893a-638a2bb9a821"). InnerVolumeSpecName "kube-api-access-kn6v9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:15:32.721279 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:15:32.721244    4521 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6843885e-1772-4f9e-893a-638a2bb9a821-maas-api-xks-ca\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:15:32.721279 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:15:32.721273    4521 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kn6v9\" (UniqueName: \"kubernetes.io/projected/6843885e-1772-4f9e-893a-638a2bb9a821-kube-api-access-kn6v9\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:15:33.428084 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:15:33.428050    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-49a6a4-q252h"
Oct 30 21:15:33.428293 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:15:33.428091    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-49a6a4-q252h" event={"ID":"6843885e-1772-4f9e-893a-638a2bb9a821","Type":"ContainerDied","Data":"82eaec1bae2ffae61f6072eef5606a4f223ec00e974a538eff245d6bb5484f19"}
Oct 30 21:15:33.428293 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:15:33.428141    4521 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="82eaec1bae2ffae61f6072eef5606a4f223ec00e974a538eff245d6bb5484f19"
Oct 30 21:15:34.718791 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:15:34.718705    4521 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-49a6a4/simulator-single-main-49a6a4-kserve-85494f98d7-4kkmg"]
Oct 30 21:15:40.476557 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:15:40.476517    4521 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-49a6a4-q252h"]
Oct 30 21:15:40.479647 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:15:40.479613    4521 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-49a6a4-q252h"]
Oct 30 21:15:40.791119 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:15:40.791041    4521 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6843885e-1772-4f9e-893a-638a2bb9a821" path="/var/lib/kubelet/pods/6843885e-1772-4f9e-893a-638a2bb9a821/volumes"
Oct 30 21:15:41.893018 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:15:41.892978    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4kxb"]
Oct 30 21:15:46.428129 ip-10-0-131-125 kubenswrapper[4521]: E1030 21:15:46.428090    4521 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lkn2k" containerName="extractor"
Oct 30 21:15:49.738724 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:15:49.738662    4521 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-49a6a4/simulator-single-main-49a6a4-kserve-85494f98d7-4kkmg" podUID="ed978d82-daa4-4d8a-90a7-a13e95243a00" containerName="main" containerID="cri-o://3c6c4609656c3d7915a7a4203a89f3f685c77a72bb666417af6e577333e8079b" gracePeriod=45
Oct 30 21:15:49.980627 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:15:49.980600    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-49a6a4/simulator-single-main-49a6a4-kserve-85494f98d7-4kkmg"
Oct 30 21:15:50.074877 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:15:50.074794    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed978d82-daa4-4d8a-90a7-a13e95243a00-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed978d82-daa4-4d8a-90a7-a13e95243a00-tmp-dir\") pod \"ed978d82-daa4-4d8a-90a7-a13e95243a00\" (UID: \"ed978d82-daa4-4d8a-90a7-a13e95243a00\") "
Oct 30 21:15:50.074877 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:15:50.074860    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed978d82-daa4-4d8a-90a7-a13e95243a00-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed978d82-daa4-4d8a-90a7-a13e95243a00-model-cache\") pod \"ed978d82-daa4-4d8a-90a7-a13e95243a00\" (UID: \"ed978d82-daa4-4d8a-90a7-a13e95243a00\") "
Oct 30 21:15:50.075096 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:15:50.074887    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed978d82-daa4-4d8a-90a7-a13e95243a00-tls-certs\" (UniqueName: \"kubernetes.io/secret/ed978d82-daa4-4d8a-90a7-a13e95243a00-tls-certs\") pod \"ed978d82-daa4-4d8a-90a7-a13e95243a00\" (UID: \"ed978d82-daa4-4d8a-90a7-a13e95243a00\") "
Oct 30 21:15:50.075096 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:15:50.074992    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed978d82-daa4-4d8a-90a7-a13e95243a00-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed978d82-daa4-4d8a-90a7-a13e95243a00-dshm\") pod \"ed978d82-daa4-4d8a-90a7-a13e95243a00\" (UID: \"ed978d82-daa4-4d8a-90a7-a13e95243a00\") "
Oct 30 21:15:50.075096 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:15:50.075026    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed978d82-daa4-4d8a-90a7-a13e95243a00-home\" (UniqueName: \"kubernetes.io/empty-dir/ed978d82-daa4-4d8a-90a7-a13e95243a00-home\") pod \"ed978d82-daa4-4d8a-90a7-a13e95243a00\" (UID: \"ed978d82-daa4-4d8a-90a7-a13e95243a00\") "
Oct 30 21:15:50.075096 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:15:50.075062    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed978d82-daa4-4d8a-90a7-a13e95243a00-kube-api-access-4h8j4\" (UniqueName: \"kubernetes.io/projected/ed978d82-daa4-4d8a-90a7-a13e95243a00-kube-api-access-4h8j4\") pod \"ed978d82-daa4-4d8a-90a7-a13e95243a00\" (UID: \"ed978d82-daa4-4d8a-90a7-a13e95243a00\") "
Oct 30 21:15:50.075320 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:15:50.075076    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed978d82-daa4-4d8a-90a7-a13e95243a00-tmp-dir" pod "ed978d82-daa4-4d8a-90a7-a13e95243a00" (UID: "ed978d82-daa4-4d8a-90a7-a13e95243a00"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:15:50.075320 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:15:50.075165    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed978d82-daa4-4d8a-90a7-a13e95243a00-model-cache" pod "ed978d82-daa4-4d8a-90a7-a13e95243a00" (UID: "ed978d82-daa4-4d8a-90a7-a13e95243a00"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:15:50.075320 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:15:50.075255    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed978d82-daa4-4d8a-90a7-a13e95243a00-home" pod "ed978d82-daa4-4d8a-90a7-a13e95243a00" (UID: "ed978d82-daa4-4d8a-90a7-a13e95243a00"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:15:50.075657 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:15:50.075397    4521 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed978d82-daa4-4d8a-90a7-a13e95243a00-home\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:15:50.075657 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:15:50.075414    4521 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed978d82-daa4-4d8a-90a7-a13e95243a00-tmp-dir\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:15:50.075657 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:15:50.075437    4521 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed978d82-daa4-4d8a-90a7-a13e95243a00-model-cache\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:15:50.077417 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:15:50.077381    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed978d82-daa4-4d8a-90a7-a13e95243a00-dshm" pod "ed978d82-daa4-4d8a-90a7-a13e95243a00" (UID: "ed978d82-daa4-4d8a-90a7-a13e95243a00"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:15:50.077505 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:15:50.077474    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed978d82-daa4-4d8a-90a7-a13e95243a00-kube-api-access-4h8j4" pod "ed978d82-daa4-4d8a-90a7-a13e95243a00" (UID: "ed978d82-daa4-4d8a-90a7-a13e95243a00"). InnerVolumeSpecName "kube-api-access-4h8j4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:15:50.077549 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:15:50.077523    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed978d82-daa4-4d8a-90a7-a13e95243a00-tls-certs" pod "ed978d82-daa4-4d8a-90a7-a13e95243a00" (UID: "ed978d82-daa4-4d8a-90a7-a13e95243a00"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:15:50.176459 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:15:50.176427    4521 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed978d82-daa4-4d8a-90a7-a13e95243a00-dshm\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:15:50.176459 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:15:50.176454    4521 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4h8j4\" (UniqueName: \"kubernetes.io/projected/ed978d82-daa4-4d8a-90a7-a13e95243a00-kube-api-access-4h8j4\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:15:50.176459 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:15:50.176463    4521 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed978d82-daa4-4d8a-90a7-a13e95243a00-tls-certs\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:15:50.518934 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:15:50.518900    4521 generic.go:356] "Generic (PLEG): container finished" podID="ed978d82-daa4-4d8a-90a7-a13e95243a00" containerID="3c6c4609656c3d7915a7a4203a89f3f685c77a72bb666417af6e577333e8079b" exitCode=0
Oct 30 21:15:50.519128 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:15:50.518948    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-49a6a4/simulator-single-main-49a6a4-kserve-85494f98d7-4kkmg" event={"ID":"ed978d82-daa4-4d8a-90a7-a13e95243a00","Type":"ContainerDied","Data":"3c6c4609656c3d7915a7a4203a89f3f685c77a72bb666417af6e577333e8079b"}
Oct 30 21:15:50.519128 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:15:50.518979    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-49a6a4/simulator-single-main-49a6a4-kserve-85494f98d7-4kkmg" event={"ID":"ed978d82-daa4-4d8a-90a7-a13e95243a00","Type":"ContainerDied","Data":"0576e92ac24fac848cc724bdac009e0f499c7adc889520cf398abf59e8a0304d"}
Oct 30 21:15:50.519128 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:15:50.518976    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-49a6a4/simulator-single-main-49a6a4-kserve-85494f98d7-4kkmg"
Oct 30 21:15:50.519128 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:15:50.518990    4521 scope.go:122] "RemoveContainer" containerID="3c6c4609656c3d7915a7a4203a89f3f685c77a72bb666417af6e577333e8079b"
Oct 30 21:15:50.528972 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:15:50.528955    4521 scope.go:122] "RemoveContainer" containerID="3c6c4609656c3d7915a7a4203a89f3f685c77a72bb666417af6e577333e8079b"
Oct 30 21:15:50.529254 ip-10-0-131-125 kubenswrapper[4521]: E1030 21:15:50.529235    4521 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c6c4609656c3d7915a7a4203a89f3f685c77a72bb666417af6e577333e8079b\": container with ID starting with 3c6c4609656c3d7915a7a4203a89f3f685c77a72bb666417af6e577333e8079b not found: ID does not exist" containerID="3c6c4609656c3d7915a7a4203a89f3f685c77a72bb666417af6e577333e8079b"
Oct 30 21:15:50.529346 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:15:50.529262    4521 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c6c4609656c3d7915a7a4203a89f3f685c77a72bb666417af6e577333e8079b"} err="failed to get container status \"3c6c4609656c3d7915a7a4203a89f3f685c77a72bb666417af6e577333e8079b\": rpc error: code = NotFound desc = could not find container \"3c6c4609656c3d7915a7a4203a89f3f685c77a72bb666417af6e577333e8079b\": container with ID starting with 3c6c4609656c3d7915a7a4203a89f3f685c77a72bb666417af6e577333e8079b not found: ID does not exist"
Oct 30 21:15:50.541538 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:15:50.541508    4521 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-49a6a4/simulator-single-main-49a6a4-kserve-85494f98d7-4kkmg"]
Oct 30 21:15:50.545656 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:15:50.545633    4521 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-49a6a4/simulator-single-main-49a6a4-kserve-85494f98d7-4kkmg"]
Oct 30 21:15:50.790870 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:15:50.790791    4521 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed978d82-daa4-4d8a-90a7-a13e95243a00" path="/var/lib/kubelet/pods/ed978d82-daa4-4d8a-90a7-a13e95243a00/volumes"
Oct 30 21:16:00.121344 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:16:00.121280    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4kxb"]
Oct 30 21:16:04.105869 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:16:04.105831    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4kxb"]
Oct 30 21:16:07.407710 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:16:07.407673    4521 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-qtt5r"]
Oct 30 21:16:07.408237 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:16:07.407931    4521 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-qtt5r" podUID="30233858-576c-450d-a5bf-f0d5ca1dd905" containerName="manager" containerID="cri-o://f361ed9ec3e0dac475abc6eb24d31fda6f2e3f65e352696eed306ec28fe47f93" gracePeriod=10
Oct 30 21:16:07.611466 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:16:07.611433    4521 generic.go:356] "Generic (PLEG): container finished" podID="30233858-576c-450d-a5bf-f0d5ca1dd905" containerID="f361ed9ec3e0dac475abc6eb24d31fda6f2e3f65e352696eed306ec28fe47f93" exitCode=0
Oct 30 21:16:07.611637 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:16:07.611511    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-qtt5r" event={"ID":"30233858-576c-450d-a5bf-f0d5ca1dd905","Type":"ContainerDied","Data":"f361ed9ec3e0dac475abc6eb24d31fda6f2e3f65e352696eed306ec28fe47f93"}
Oct 30 21:16:07.653785 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:16:07.653761    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-qtt5r"
Oct 30 21:16:07.834179 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:16:07.834144    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30233858-576c-450d-a5bf-f0d5ca1dd905-kube-api-access-2rclh\" (UniqueName: \"kubernetes.io/projected/30233858-576c-450d-a5bf-f0d5ca1dd905-kube-api-access-2rclh\") pod \"30233858-576c-450d-a5bf-f0d5ca1dd905\" (UID: \"30233858-576c-450d-a5bf-f0d5ca1dd905\") "
Oct 30 21:16:07.834415 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:16:07.834221    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30233858-576c-450d-a5bf-f0d5ca1dd905-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/30233858-576c-450d-a5bf-f0d5ca1dd905-extensions-socket-volume\") pod \"30233858-576c-450d-a5bf-f0d5ca1dd905\" (UID: \"30233858-576c-450d-a5bf-f0d5ca1dd905\") "
Oct 30 21:16:07.834671 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:16:07.834646    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30233858-576c-450d-a5bf-f0d5ca1dd905-extensions-socket-volume" pod "30233858-576c-450d-a5bf-f0d5ca1dd905" (UID: "30233858-576c-450d-a5bf-f0d5ca1dd905"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:16:07.836284 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:16:07.836262    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30233858-576c-450d-a5bf-f0d5ca1dd905-kube-api-access-2rclh" pod "30233858-576c-450d-a5bf-f0d5ca1dd905" (UID: "30233858-576c-450d-a5bf-f0d5ca1dd905"). InnerVolumeSpecName "kube-api-access-2rclh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:16:07.934978 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:16:07.934940    4521 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rclh\" (UniqueName: \"kubernetes.io/projected/30233858-576c-450d-a5bf-f0d5ca1dd905-kube-api-access-2rclh\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:16:07.934978 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:16:07.934973    4521 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/30233858-576c-450d-a5bf-f0d5ca1dd905-extensions-socket-volume\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:16:08.617634 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:16:08.617584    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-qtt5r" event={"ID":"30233858-576c-450d-a5bf-f0d5ca1dd905","Type":"ContainerDied","Data":"d4ee57970d9e132201a39bf2827575b3f832c3527a77c64c3cb64d0b725b1f18"}
Oct 30 21:16:08.618161 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:16:08.617634    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-qtt5r"
Oct 30 21:16:08.618161 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:16:08.617681    4521 scope.go:122] "RemoveContainer" containerID="f361ed9ec3e0dac475abc6eb24d31fda6f2e3f65e352696eed306ec28fe47f93"
Oct 30 21:16:08.643973 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:16:08.643944    4521 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-qtt5r"]
Oct 30 21:16:08.655428 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:16:08.655401    4521 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-qtt5r"]
Oct 30 21:16:08.790745 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:16:08.790701    4521 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30233858-576c-450d-a5bf-f0d5ca1dd905" path="/var/lib/kubelet/pods/30233858-576c-450d-a5bf-f0d5ca1dd905/volumes"
Oct 30 21:16:16.427878 ip-10-0-131-125 kubenswrapper[4521]: E1030 21:16:16.427843    4521 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lkn2k" containerName="extractor"
Oct 30 21:16:26.135532 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:16:26.135497    4521 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-fw7lj"]
Oct 30 21:16:26.135982 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:16:26.135821    4521 cpu_manager.go:414] "containerMap: removing container" podUID="30233858-576c-450d-a5bf-f0d5ca1dd905" containerName="manager"
Oct 30 21:16:26.135982 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:16:26.135831    4521 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30233858-576c-450d-a5bf-f0d5ca1dd905" containerName="manager"
Oct 30 21:16:26.135982 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:16:26.135841    4521 cpu_manager.go:414] "containerMap: removing container" podUID="6843885e-1772-4f9e-893a-638a2bb9a821" containerName="revoke-keys"
Oct 30 21:16:26.135982 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:16:26.135847    4521 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6843885e-1772-4f9e-893a-638a2bb9a821" containerName="revoke-keys"
Oct 30 21:16:26.135982 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:16:26.135870    4521 cpu_manager.go:414] "containerMap: removing container" podUID="ed978d82-daa4-4d8a-90a7-a13e95243a00" containerName="main"
Oct 30 21:16:26.135982 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:16:26.135875    4521 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed978d82-daa4-4d8a-90a7-a13e95243a00" containerName="main"
Oct 30 21:16:26.135982 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:16:26.135936    4521 memory_manager.go:365] "RemoveStaleState removing state" podUID="30233858-576c-450d-a5bf-f0d5ca1dd905" containerName="manager"
Oct 30 21:16:26.135982 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:16:26.135945    4521 memory_manager.go:365] "RemoveStaleState removing state" podUID="6843885e-1772-4f9e-893a-638a2bb9a821" containerName="revoke-keys"
Oct 30 21:16:26.135982 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:16:26.135955    4521 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed978d82-daa4-4d8a-90a7-a13e95243a00" containerName="main"
Oct 30 21:16:26.139014 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:16:26.138985    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-fw7lj"
Oct 30 21:16:26.156452 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:16:26.156419    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-fw7lj"]
Oct 30 21:16:26.181226 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:16:26.181192    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5311e7d9-ac35-419c-a7d8-2bceba0f4f44-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-fw7lj\" (UID: \"5311e7d9-ac35-419c-a7d8-2bceba0f4f44\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-fw7lj"
Oct 30 21:16:26.181469 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:16:26.181238    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5311e7d9-ac35-419c-a7d8-2bceba0f4f44-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-fw7lj\" (UID: \"5311e7d9-ac35-419c-a7d8-2bceba0f4f44\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-fw7lj"
Oct 30 21:16:26.181469 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:16:26.181379    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5311e7d9-ac35-419c-a7d8-2bceba0f4f44-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-fw7lj\" (UID: \"5311e7d9-ac35-419c-a7d8-2bceba0f4f44\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-fw7lj"
Oct 30 21:16:26.181469 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:16:26.181407    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5311e7d9-ac35-419c-a7d8-2bceba0f4f44-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-fw7lj\" (UID: \"5311e7d9-ac35-419c-a7d8-2bceba0f4f44\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-fw7lj"
Oct 30 21:16:26.181469 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:16:26.181434    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlrv7\" (UniqueName: \"kubernetes.io/projected/5311e7d9-ac35-419c-a7d8-2bceba0f4f44-kube-api-access-qlrv7\") pod \"maas-default-gateway-openshift-default-58c6585447-fw7lj\" (UID: \"5311e7d9-ac35-419c-a7d8-2bceba0f4f44\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-fw7lj"
Oct 30 21:16:26.181665 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:16:26.181475    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5311e7d9-ac35-419c-a7d8-2bceba0f4f44-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-fw7lj\" (UID: \"5311e7d9-ac35-419c-a7d8-2bceba0f4f44\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-fw7lj"
Oct 30 21:16:26.181665 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:16:26.181551    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5311e7d9-ac35-419c-a7d8-2bceba0f4f44-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-fw7lj\" (UID: \"5311e7d9-ac35-419c-a7d8-2bceba0f4f44\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-fw7lj"
Oct 30 21:16:26.181665 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:16:26.181650    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5311e7d9-ac35-419c-a7d8-2bceba0f4f44-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-fw7lj\" (UID: \"5311e7d9-ac35-419c-a7d8-2bceba0f4f44\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-fw7lj"
Oct 30 21:16:26.181775 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:16:26.181690    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5311e7d9-ac35-419c-a7d8-2bceba0f4f44-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-fw7lj\" (UID: \"5311e7d9-ac35-419c-a7d8-2bceba0f4f44\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-fw7lj"
Oct 30 21:16:26.282218 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:16:26.282175    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5311e7d9-ac35-419c-a7d8-2bceba0f4f44-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-fw7lj\" (UID: \"5311e7d9-ac35-419c-a7d8-2bceba0f4f44\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-fw7lj"
Oct 30 21:16:26.282463 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:16:26.282225    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5311e7d9-ac35-419c-a7d8-2bceba0f4f44-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-fw7lj\" (UID: \"5311e7d9-ac35-419c-a7d8-2bceba0f4f44\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-fw7lj"
Oct 30 21:16:26.282463 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:16:26.282255    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlrv7\" (UniqueName: \"kubernetes.io/projected/5311e7d9-ac35-419c-a7d8-2bceba0f4f44-kube-api-access-qlrv7\") pod \"maas-default-gateway-openshift-default-58c6585447-fw7lj\" (UID: \"5311e7d9-ac35-419c-a7d8-2bceba0f4f44\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-fw7lj"
Oct 30 21:16:26.282463 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:16:26.282281    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5311e7d9-ac35-419c-a7d8-2bceba0f4f44-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-fw7lj\" (UID: \"5311e7d9-ac35-419c-a7d8-2bceba0f4f44\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-fw7lj"
Oct 30 21:16:26.282463 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:16:26.282306    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5311e7d9-ac35-419c-a7d8-2bceba0f4f44-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-fw7lj\" (UID: \"5311e7d9-ac35-419c-a7d8-2bceba0f4f44\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-fw7lj"
Oct 30 21:16:26.282463 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:16:26.282378    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5311e7d9-ac35-419c-a7d8-2bceba0f4f44-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-fw7lj\" (UID: \"5311e7d9-ac35-419c-a7d8-2bceba0f4f44\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-fw7lj"
Oct 30 21:16:26.282463 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:16:26.282427    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5311e7d9-ac35-419c-a7d8-2bceba0f4f44-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-fw7lj\" (UID: \"5311e7d9-ac35-419c-a7d8-2bceba0f4f44\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-fw7lj"
Oct 30 21:16:26.282463 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:16:26.282449    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5311e7d9-ac35-419c-a7d8-2bceba0f4f44-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-fw7lj\" (UID: \"5311e7d9-ac35-419c-a7d8-2bceba0f4f44\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-fw7lj"
Oct 30 21:16:26.282837 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:16:26.282474    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5311e7d9-ac35-419c-a7d8-2bceba0f4f44-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-fw7lj\" (UID: \"5311e7d9-ac35-419c-a7d8-2bceba0f4f44\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-fw7lj"
Oct 30 21:16:26.282837 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:16:26.282680    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5311e7d9-ac35-419c-a7d8-2bceba0f4f44-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-fw7lj\" (UID: \"5311e7d9-ac35-419c-a7d8-2bceba0f4f44\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-fw7lj"
Oct 30 21:16:26.282953 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:16:26.282927    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5311e7d9-ac35-419c-a7d8-2bceba0f4f44-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-fw7lj\" (UID: \"5311e7d9-ac35-419c-a7d8-2bceba0f4f44\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-fw7lj"
Oct 30 21:16:26.283035 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:16:26.283010    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5311e7d9-ac35-419c-a7d8-2bceba0f4f44-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-fw7lj\" (UID: \"5311e7d9-ac35-419c-a7d8-2bceba0f4f44\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-fw7lj"
Oct 30 21:16:26.283070 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:16:26.283034    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5311e7d9-ac35-419c-a7d8-2bceba0f4f44-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-fw7lj\" (UID: \"5311e7d9-ac35-419c-a7d8-2bceba0f4f44\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-fw7lj"
Oct 30 21:16:26.283108 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:16:26.283075    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5311e7d9-ac35-419c-a7d8-2bceba0f4f44-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-fw7lj\" (UID: \"5311e7d9-ac35-419c-a7d8-2bceba0f4f44\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-fw7lj"
Oct 30 21:16:26.284614 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:16:26.284589    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5311e7d9-ac35-419c-a7d8-2bceba0f4f44-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-fw7lj\" (UID: \"5311e7d9-ac35-419c-a7d8-2bceba0f4f44\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-fw7lj"
Oct 30 21:16:26.284848 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:16:26.284829    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5311e7d9-ac35-419c-a7d8-2bceba0f4f44-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-fw7lj\" (UID: \"5311e7d9-ac35-419c-a7d8-2bceba0f4f44\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-fw7lj"
Oct 30 21:16:26.294562 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:16:26.294534    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5311e7d9-ac35-419c-a7d8-2bceba0f4f44-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-fw7lj\" (UID: \"5311e7d9-ac35-419c-a7d8-2bceba0f4f44\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-fw7lj"
Oct 30 21:16:26.294821 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:16:26.294805    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlrv7\" (UniqueName: \"kubernetes.io/projected/5311e7d9-ac35-419c-a7d8-2bceba0f4f44-kube-api-access-qlrv7\") pod \"maas-default-gateway-openshift-default-58c6585447-fw7lj\" (UID: \"5311e7d9-ac35-419c-a7d8-2bceba0f4f44\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-fw7lj"
Oct 30 21:16:26.452639 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:16:26.452604    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-fw7lj"
Oct 30 21:16:26.736735 ip-10-0-131-125 kubenswrapper[4521]: W1030 21:16:26.736703    4521 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5311e7d9_ac35_419c_a7d8_2bceba0f4f44.slice/crio-10adce113004b5d05bc9cf25c3f496fefa49474c5a0c76e81cb834b09bf3ff19 WatchSource:0}: Error finding container 10adce113004b5d05bc9cf25c3f496fefa49474c5a0c76e81cb834b09bf3ff19: Status 404 returned error can't find the container with id 10adce113004b5d05bc9cf25c3f496fefa49474c5a0c76e81cb834b09bf3ff19
Oct 30 21:16:26.736871 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:16:26.736849    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-fw7lj"]
Oct 30 21:16:26.738909 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:16:26.738876    4521 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:16:26.739039 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:16:26.738946    4521 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:16:26.739039 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:16:26.739000    4521 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:16:27.716830 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:16:27.716794    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-fw7lj" event={"ID":"5311e7d9-ac35-419c-a7d8-2bceba0f4f44","Type":"ContainerStarted","Data":"f40440d051f8d83d4435c15958ef0eb691711b55b5c5ed6f47d8912b00c7cf3c"}
Oct 30 21:16:27.717229 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:16:27.716851    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-fw7lj" event={"ID":"5311e7d9-ac35-419c-a7d8-2bceba0f4f44","Type":"ContainerStarted","Data":"10adce113004b5d05bc9cf25c3f496fefa49474c5a0c76e81cb834b09bf3ff19"}
Oct 30 21:16:27.742422 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:16:27.742371    4521 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-fw7lj" podStartSLOduration=1.742357879 podStartE2EDuration="1.742357879s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:16:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:16:27.740295135 +0000 UTC m=+2255.085916372" watchObservedRunningTime="2026-10-30 21:16:27.742357879 +0000 UTC m=+2255.087979114"
Oct 30 21:16:28.452986 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:16:28.452948    4521 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-fw7lj"
Oct 30 21:16:28.457749 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:16:28.457716    4521 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-fw7lj"
Oct 30 21:16:28.721948 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:16:28.721869    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-fw7lj"
Oct 30 21:16:28.722821 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:16:28.722800    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-fw7lj"
Oct 30 21:16:41.152434 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:16:41.152394    4521 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-mgx2j"]
Oct 30 21:16:41.155988 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:16:41.155969    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-mgx2j"
Oct 30 21:16:41.173280 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:16:41.173251    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-mgx2j"]
Oct 30 21:16:41.216577 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:16:41.216539    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f51564c-e79a-437a-9e62-3b2750e0916a-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-mgx2j\" (UID: \"3f51564c-e79a-437a-9e62-3b2750e0916a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-mgx2j"
Oct 30 21:16:41.216760 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:16:41.216591    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f51564c-e79a-437a-9e62-3b2750e0916a-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-mgx2j\" (UID: \"3f51564c-e79a-437a-9e62-3b2750e0916a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-mgx2j"
Oct 30 21:16:41.216760 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:16:41.216655    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f51564c-e79a-437a-9e62-3b2750e0916a-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-mgx2j\" (UID: \"3f51564c-e79a-437a-9e62-3b2750e0916a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-mgx2j"
Oct 30 21:16:41.216760 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:16:41.216730    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f51564c-e79a-437a-9e62-3b2750e0916a-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-mgx2j\" (UID: \"3f51564c-e79a-437a-9e62-3b2750e0916a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-mgx2j"
Oct 30 21:16:41.216923 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:16:41.216900    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f51564c-e79a-437a-9e62-3b2750e0916a-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-mgx2j\" (UID: \"3f51564c-e79a-437a-9e62-3b2750e0916a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-mgx2j"
Oct 30 21:16:41.216963 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:16:41.216933    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f51564c-e79a-437a-9e62-3b2750e0916a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-mgx2j\" (UID: \"3f51564c-e79a-437a-9e62-3b2750e0916a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-mgx2j"
Oct 30 21:16:41.217007 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:16:41.216997    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6s5t\" (UniqueName: \"kubernetes.io/projected/3f51564c-e79a-437a-9e62-3b2750e0916a-kube-api-access-c6s5t\") pod \"maas-default-gateway-openshift-default-58c6585447-mgx2j\" (UID: \"3f51564c-e79a-437a-9e62-3b2750e0916a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-mgx2j"
Oct 30 21:16:41.217048 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:16:41.217029    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f51564c-e79a-437a-9e62-3b2750e0916a-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-mgx2j\" (UID: \"3f51564c-e79a-437a-9e62-3b2750e0916a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-mgx2j"
Oct 30 21:16:41.217048 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:16:41.217045    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f51564c-e79a-437a-9e62-3b2750e0916a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-mgx2j\" (UID: \"3f51564c-e79a-437a-9e62-3b2750e0916a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-mgx2j"
Oct 30 21:16:41.317919 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:16:41.317882    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f51564c-e79a-437a-9e62-3b2750e0916a-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-mgx2j\" (UID: \"3f51564c-e79a-437a-9e62-3b2750e0916a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-mgx2j"
Oct 30 21:16:41.318113 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:16:41.317942    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f51564c-e79a-437a-9e62-3b2750e0916a-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-mgx2j\" (UID: \"3f51564c-e79a-437a-9e62-3b2750e0916a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-mgx2j"
Oct 30 21:16:41.318113 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:16:41.317970    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f51564c-e79a-437a-9e62-3b2750e0916a-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-mgx2j\" (UID: \"3f51564c-e79a-437a-9e62-3b2750e0916a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-mgx2j"
Oct 30 21:16:41.318113 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:16:41.318008    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f51564c-e79a-437a-9e62-3b2750e0916a-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-mgx2j\" (UID: \"3f51564c-e79a-437a-9e62-3b2750e0916a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-mgx2j"
Oct 30 21:16:41.318113 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:16:41.318037    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f51564c-e79a-437a-9e62-3b2750e0916a-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-mgx2j\" (UID: \"3f51564c-e79a-437a-9e62-3b2750e0916a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-mgx2j"
Oct 30 21:16:41.318302 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:16:41.318165    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f51564c-e79a-437a-9e62-3b2750e0916a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-mgx2j\" (UID: \"3f51564c-e79a-437a-9e62-3b2750e0916a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-mgx2j"
Oct 30 21:16:41.318302 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:16:41.318237    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6s5t\" (UniqueName: \"kubernetes.io/projected/3f51564c-e79a-437a-9e62-3b2750e0916a-kube-api-access-c6s5t\") pod \"maas-default-gateway-openshift-default-58c6585447-mgx2j\" (UID: \"3f51564c-e79a-437a-9e62-3b2750e0916a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-mgx2j"
Oct 30 21:16:41.318465 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:16:41.318306    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f51564c-e79a-437a-9e62-3b2750e0916a-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-mgx2j\" (UID: \"3f51564c-e79a-437a-9e62-3b2750e0916a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-mgx2j"
Oct 30 21:16:41.318465 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:16:41.318313    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f51564c-e79a-437a-9e62-3b2750e0916a-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-mgx2j\" (UID: \"3f51564c-e79a-437a-9e62-3b2750e0916a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-mgx2j"
Oct 30 21:16:41.318465 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:16:41.318408    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f51564c-e79a-437a-9e62-3b2750e0916a-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-mgx2j\" (UID: \"3f51564c-e79a-437a-9e62-3b2750e0916a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-mgx2j"
Oct 30 21:16:41.318465 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:16:41.318428    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f51564c-e79a-437a-9e62-3b2750e0916a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-mgx2j\" (UID: \"3f51564c-e79a-437a-9e62-3b2750e0916a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-mgx2j"
Oct 30 21:16:41.318683 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:16:41.318580    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f51564c-e79a-437a-9e62-3b2750e0916a-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-mgx2j\" (UID: \"3f51564c-e79a-437a-9e62-3b2750e0916a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-mgx2j"
Oct 30 21:16:41.318789 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:16:41.318770    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f51564c-e79a-437a-9e62-3b2750e0916a-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-mgx2j\" (UID: \"3f51564c-e79a-437a-9e62-3b2750e0916a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-mgx2j"
Oct 30 21:16:41.319139 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:16:41.319122    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f51564c-e79a-437a-9e62-3b2750e0916a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-mgx2j\" (UID: \"3f51564c-e79a-437a-9e62-3b2750e0916a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-mgx2j"
Oct 30 21:16:41.320287 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:16:41.320271    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f51564c-e79a-437a-9e62-3b2750e0916a-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-mgx2j\" (UID: \"3f51564c-e79a-437a-9e62-3b2750e0916a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-mgx2j"
Oct 30 21:16:41.320664 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:16:41.320646    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f51564c-e79a-437a-9e62-3b2750e0916a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-mgx2j\" (UID: \"3f51564c-e79a-437a-9e62-3b2750e0916a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-mgx2j"
Oct 30 21:16:41.331242 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:16:41.331217    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f51564c-e79a-437a-9e62-3b2750e0916a-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-mgx2j\" (UID: \"3f51564c-e79a-437a-9e62-3b2750e0916a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-mgx2j"
Oct 30 21:16:41.331532 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:16:41.331511    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6s5t\" (UniqueName: \"kubernetes.io/projected/3f51564c-e79a-437a-9e62-3b2750e0916a-kube-api-access-c6s5t\") pod \"maas-default-gateway-openshift-default-58c6585447-mgx2j\" (UID: \"3f51564c-e79a-437a-9e62-3b2750e0916a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-mgx2j"
Oct 30 21:16:41.468864 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:16:41.468820    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-mgx2j"
Oct 30 21:16:41.751476 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:16:41.751447    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-mgx2j"]
Oct 30 21:16:41.752816 ip-10-0-131-125 kubenswrapper[4521]: W1030 21:16:41.752789    4521 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f51564c_e79a_437a_9e62_3b2750e0916a.slice/crio-2f4ac23f88ee8ae1a6cd26533e109c957be0c306657fe3842fbc1218bb78246a WatchSource:0}: Error finding container 2f4ac23f88ee8ae1a6cd26533e109c957be0c306657fe3842fbc1218bb78246a: Status 404 returned error can't find the container with id 2f4ac23f88ee8ae1a6cd26533e109c957be0c306657fe3842fbc1218bb78246a
Oct 30 21:16:41.755005 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:16:41.754964    4521 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:16:41.755094 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:16:41.755036    4521 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:16:41.755094 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:16:41.755064    4521 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:16:41.796934 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:16:41.796900    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-mgx2j" event={"ID":"3f51564c-e79a-437a-9e62-3b2750e0916a","Type":"ContainerStarted","Data":"2f4ac23f88ee8ae1a6cd26533e109c957be0c306657fe3842fbc1218bb78246a"}
Oct 30 21:16:42.803103 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:16:42.803060    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-mgx2j" event={"ID":"3f51564c-e79a-437a-9e62-3b2750e0916a","Type":"ContainerStarted","Data":"435ca0c39a28f68b76e87a573806366d1f3658a529bd5f9328f72cf6cddb55e9"}
Oct 30 21:16:42.830853 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:16:42.830801    4521 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-mgx2j" podStartSLOduration=1.830788702 podStartE2EDuration="1.830788702s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:16:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:16:42.829263114 +0000 UTC m=+2270.174884381" watchObservedRunningTime="2026-10-30 21:16:42.830788702 +0000 UTC m=+2270.176409940"
Oct 30 21:16:43.469034 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:16:43.468995    4521 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-mgx2j"
Oct 30 21:16:43.474223 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:16:43.474198    4521 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-mgx2j"
Oct 30 21:16:43.808540 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:16:43.808453    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-mgx2j"
Oct 30 21:16:43.809652 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:16:43.809631    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-mgx2j"
Oct 30 21:16:46.428222 ip-10-0-131-125 kubenswrapper[4521]: E1030 21:16:46.428188    4521 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lkn2k" containerName="extractor"
Oct 30 21:17:16.428923 ip-10-0-131-125 kubenswrapper[4521]: E1030 21:17:16.428832    4521 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lkn2k" containerName="extractor"
Oct 30 21:17:32.301354 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:17:32.301301    4521 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-2zlw7"]
Oct 30 21:17:32.304277 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:17:32.304260    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-2zlw7"
Oct 30 21:17:32.310701 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:17:32.310670    4521 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-lpzsx\""
Oct 30 21:17:32.321942 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:17:32.321909    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-2zlw7"]
Oct 30 21:17:32.375249 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:17:32.375213    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0c31240f-bde3-4853-a2c3-6f83ab926529-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-2zlw7\" (UID: \"0c31240f-bde3-4853-a2c3-6f83ab926529\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-2zlw7"
Oct 30 21:17:32.375462 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:17:32.375262    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjnwh\" (UniqueName: \"kubernetes.io/projected/0c31240f-bde3-4853-a2c3-6f83ab926529-kube-api-access-vjnwh\") pod \"kuadrant-operator-controller-manager-678d87766f-2zlw7\" (UID: \"0c31240f-bde3-4853-a2c3-6f83ab926529\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-2zlw7"
Oct 30 21:17:32.476423 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:17:32.476386    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0c31240f-bde3-4853-a2c3-6f83ab926529-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-2zlw7\" (UID: \"0c31240f-bde3-4853-a2c3-6f83ab926529\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-2zlw7"
Oct 30 21:17:32.476611 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:17:32.476441    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjnwh\" (UniqueName: \"kubernetes.io/projected/0c31240f-bde3-4853-a2c3-6f83ab926529-kube-api-access-vjnwh\") pod \"kuadrant-operator-controller-manager-678d87766f-2zlw7\" (UID: \"0c31240f-bde3-4853-a2c3-6f83ab926529\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-2zlw7"
Oct 30 21:17:32.476869 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:17:32.476847    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0c31240f-bde3-4853-a2c3-6f83ab926529-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-2zlw7\" (UID: \"0c31240f-bde3-4853-a2c3-6f83ab926529\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-2zlw7"
Oct 30 21:17:32.489115 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:17:32.489087    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjnwh\" (UniqueName: \"kubernetes.io/projected/0c31240f-bde3-4853-a2c3-6f83ab926529-kube-api-access-vjnwh\") pod \"kuadrant-operator-controller-manager-678d87766f-2zlw7\" (UID: \"0c31240f-bde3-4853-a2c3-6f83ab926529\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-2zlw7"
Oct 30 21:17:32.617353 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:17:32.617244    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-2zlw7"
Oct 30 21:17:32.933123 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:17:32.933096    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-2zlw7"]
Oct 30 21:17:32.935265 ip-10-0-131-125 kubenswrapper[4521]: W1030 21:17:32.935236    4521 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c31240f_bde3_4853_a2c3_6f83ab926529.slice/crio-fcba26a62656766fc4b231392226d2f6b92bb68e9ac2d8669b8135de7a7840c5 WatchSource:0}: Error finding container fcba26a62656766fc4b231392226d2f6b92bb68e9ac2d8669b8135de7a7840c5: Status 404 returned error can't find the container with id fcba26a62656766fc4b231392226d2f6b92bb68e9ac2d8669b8135de7a7840c5
Oct 30 21:17:33.050589 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:17:33.050556    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-2zlw7" event={"ID":"0c31240f-bde3-4853-a2c3-6f83ab926529","Type":"ContainerStarted","Data":"512d09d8ce1de274f51fe3bff6fe4f86dbb7180f50306b2e398f458f95bbfe99"}
Oct 30 21:17:33.050780 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:17:33.050610    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-2zlw7" event={"ID":"0c31240f-bde3-4853-a2c3-6f83ab926529","Type":"ContainerStarted","Data":"fcba26a62656766fc4b231392226d2f6b92bb68e9ac2d8669b8135de7a7840c5"}
Oct 30 21:17:33.050780 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:17:33.050654    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-2zlw7"
Oct 30 21:17:33.075378 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:17:33.075292    4521 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-2zlw7" podStartSLOduration=1.075274828 podStartE2EDuration="1.075274828s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:17:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:17:33.074626481 +0000 UTC m=+2320.420247719" watchObservedRunningTime="2026-10-30 21:17:33.075274828 +0000 UTC m=+2320.420896079"
Oct 30 21:17:44.057537 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:17:44.057502    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-2zlw7"
Oct 30 21:17:46.428820 ip-10-0-131-125 kubenswrapper[4521]: E1030 21:17:46.428780    4521 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lkn2k" containerName="extractor"
Oct 30 21:17:51.207718 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:17:51.207681    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4kxb"]
Oct 30 21:17:59.764547 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:17:59.764496    4521 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-fw7lj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:17:59.764998 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:17:59.764594    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-fw7lj" podUID="5311e7d9-ac35-419c-a7d8-2bceba0f4f44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:17:59.849520 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:17:59.849478    4521 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-mgx2j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:17:59.849706 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:17:59.849556    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-mgx2j" podUID="3f51564c-e79a-437a-9e62-3b2750e0916a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:18:01.285868 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:18:01.285777    4521 generic.go:356] "Generic (PLEG): container finished" podID="5311e7d9-ac35-419c-a7d8-2bceba0f4f44" containerID="f40440d051f8d83d4435c15958ef0eb691711b55b5c5ed6f47d8912b00c7cf3c" exitCode=0
Oct 30 21:18:01.285868 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:18:01.285854    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-fw7lj" event={"ID":"5311e7d9-ac35-419c-a7d8-2bceba0f4f44","Type":"ContainerDied","Data":"f40440d051f8d83d4435c15958ef0eb691711b55b5c5ed6f47d8912b00c7cf3c"}
Oct 30 21:18:01.286431 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:18:01.286256    4521 scope.go:122] "RemoveContainer" containerID="f40440d051f8d83d4435c15958ef0eb691711b55b5c5ed6f47d8912b00c7cf3c"
Oct 30 21:18:01.287611 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:18:01.287578    4521 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:18:01.287743 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:18:01.287660    4521 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:18:01.287743 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:18:01.287681    4521 generic.go:356] "Generic (PLEG): container finished" podID="3f51564c-e79a-437a-9e62-3b2750e0916a" containerID="435ca0c39a28f68b76e87a573806366d1f3658a529bd5f9328f72cf6cddb55e9" exitCode=0
Oct 30 21:18:01.287743 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:18:01.287696    4521 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:18:01.287849 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:18:01.287752    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-mgx2j" event={"ID":"3f51564c-e79a-437a-9e62-3b2750e0916a","Type":"ContainerDied","Data":"435ca0c39a28f68b76e87a573806366d1f3658a529bd5f9328f72cf6cddb55e9"}
Oct 30 21:18:01.288058 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:18:01.288043    4521 scope.go:122] "RemoveContainer" containerID="435ca0c39a28f68b76e87a573806366d1f3658a529bd5f9328f72cf6cddb55e9"
Oct 30 21:18:01.288993 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:18:01.288965    4521 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:18:01.289074 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:18:01.289030    4521 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:18:01.289114 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:18:01.289082    4521 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:18:02.294581 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:18:02.294545    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-fw7lj" event={"ID":"5311e7d9-ac35-419c-a7d8-2bceba0f4f44","Type":"ContainerStarted","Data":"b7e1ff9e2644bae248b3592af2edc3f3ca0d5ba4a8fa1978ed761e349a2eaf4c"}
Oct 30 21:18:02.296520 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:18:02.296496    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-mgx2j" event={"ID":"3f51564c-e79a-437a-9e62-3b2750e0916a","Type":"ContainerStarted","Data":"2a0d1ecb7c8e7ce356d37084b84caee8bfde95aba7fd497d7af1e73e880e8f0a"}
Oct 30 21:18:02.456823 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:18:02.456785    4521 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-fw7lj"
Oct 30 21:18:02.457226 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:18:02.457136    4521 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-fw7lj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body=
Oct 30 21:18:02.457425 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:18:02.457403    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-fw7lj" podUID="5311e7d9-ac35-419c-a7d8-2bceba0f4f44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused"
Oct 30 21:18:02.469521 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:18:02.469491    4521 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-mgx2j"
Oct 30 21:18:02.469839 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:18:02.469820    4521 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-mgx2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body=
Oct 30 21:18:02.469957 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:18:02.469939    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-mgx2j" podUID="3f51564c-e79a-437a-9e62-3b2750e0916a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused"
Oct 30 21:18:03.459348 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:18:03.458794    4521 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-fw7lj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body=
Oct 30 21:18:03.459348 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:18:03.458859    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-fw7lj" podUID="5311e7d9-ac35-419c-a7d8-2bceba0f4f44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused"
Oct 30 21:18:03.469629 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:18:03.469603    4521 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-mgx2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body=
Oct 30 21:18:03.469837 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:18:03.469811    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-mgx2j" podUID="3f51564c-e79a-437a-9e62-3b2750e0916a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused"
Oct 30 21:18:04.453845 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:18:04.453800    4521 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-fw7lj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body=
Oct 30 21:18:04.454151 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:18:04.454124    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-fw7lj" podUID="5311e7d9-ac35-419c-a7d8-2bceba0f4f44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused"
Oct 30 21:18:04.469681 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:18:04.469648    4521 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-mgx2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body=
Oct 30 21:18:04.470073 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:18:04.469692    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-mgx2j" podUID="3f51564c-e79a-437a-9e62-3b2750e0916a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused"
Oct 30 21:18:05.458809 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:18:05.458762    4521 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-fw7lj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body=
Oct 30 21:18:05.459010 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:18:05.458834    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-fw7lj" podUID="5311e7d9-ac35-419c-a7d8-2bceba0f4f44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused"
Oct 30 21:18:05.469686 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:18:05.469652    4521 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-mgx2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body=
Oct 30 21:18:05.470043 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:18:05.469697    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-mgx2j" podUID="3f51564c-e79a-437a-9e62-3b2750e0916a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused"
Oct 30 21:18:06.464370 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:18:06.462645    4521 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-fw7lj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body=
Oct 30 21:18:06.464370 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:18:06.462717    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-fw7lj" podUID="5311e7d9-ac35-419c-a7d8-2bceba0f4f44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused"
Oct 30 21:18:06.469266 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:18:06.469236    4521 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-mgx2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body=
Oct 30 21:18:06.469408 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:18:06.469287    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-mgx2j" podUID="3f51564c-e79a-437a-9e62-3b2750e0916a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused"
Oct 30 21:18:07.453504 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:18:07.453456    4521 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-fw7lj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body=
Oct 30 21:18:07.453916 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:18:07.453531    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-fw7lj" podUID="5311e7d9-ac35-419c-a7d8-2bceba0f4f44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused"
Oct 30 21:18:07.470259 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:18:07.470225    4521 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-mgx2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body=
Oct 30 21:18:07.470381 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:18:07.470293    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-mgx2j" podUID="3f51564c-e79a-437a-9e62-3b2750e0916a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused"
Oct 30 21:18:08.453911 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:18:08.453868    4521 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-fw7lj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body=
Oct 30 21:18:08.454349 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:18:08.453939    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-fw7lj" podUID="5311e7d9-ac35-419c-a7d8-2bceba0f4f44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused"
Oct 30 21:18:08.469651 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:18:08.469616    4521 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-mgx2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body=
Oct 30 21:18:08.469795 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:18:08.469685    4521 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-mgx2j" podUID="3f51564c-e79a-437a-9e62-3b2750e0916a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused"
Oct 30 21:18:09.644553 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:18:09.644514    4521 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-mgx2j"
Oct 30 21:18:09.645379 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:18:09.645356    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-mgx2j"
Oct 30 21:18:09.646053 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:18:09.646028    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-mgx2j"
Oct 30 21:18:09.692517 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:18:09.692474    4521 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-fw7lj"
Oct 30 21:18:09.692929 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:18:09.692909    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-fw7lj"
Oct 30 21:18:09.693742 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:18:09.693719    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-fw7lj"
Oct 30 21:18:12.023339 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:18:12.023293    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4kxb"]
Oct 30 21:18:16.281136 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:18:16.281085    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4kxb"]
Oct 30 21:18:16.499415 ip-10-0-131-125 kubenswrapper[4521]: E1030 21:18:16.499363    4521 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lkn2k" containerName="extractor"
Oct 30 21:18:31.220231 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:18:31.220174    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4kxb"]
Oct 30 21:18:39.199746 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:18:39.199703    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4kxb"]
Oct 30 21:18:46.430464 ip-10-0-131-125 kubenswrapper[4521]: E1030 21:18:46.430422    4521 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lkn2k" containerName="extractor"
Oct 30 21:18:48.446545 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:18:48.446504    4521 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-11b629/simulator-single-main-11b629-kserve-754b5695d5-vg55f"]
Oct 30 21:18:48.450474 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:18:48.450448    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-11b629/simulator-single-main-11b629-kserve-754b5695d5-vg55f"
Oct 30 21:18:48.459138 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:18:48.458708    4521 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-11b629\"/\"openshift-service-ca.crt\""
Oct 30 21:18:48.459862 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:18:48.459827    4521 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-11b629\"/\"kube-root-ca.crt\""
Oct 30 21:18:48.462178 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:18:48.462155    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-11b629/simulator-single-main-11b629-kserve-754b5695d5-vg55f"]
Oct 30 21:18:48.477290 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:18:48.477265    4521 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-11b629\"/\"default-dockercfg-z9hnf\""
Oct 30 21:18:48.477476 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:18:48.477401    4521 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-11b629\"/\"simulator-single-main-11b629-kserve-self-signed-certs\""
Oct 30 21:18:48.631153 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:18:48.631117    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzn5t\" (UniqueName: \"kubernetes.io/projected/0d8e0e5b-a716-4dd5-b2d6-8a491fe24065-kube-api-access-zzn5t\") pod \"simulator-single-main-11b629-kserve-754b5695d5-vg55f\" (UID: \"0d8e0e5b-a716-4dd5-b2d6-8a491fe24065\") " pod="e2e-models-e2e-worker-single-main-11b629/simulator-single-main-11b629-kserve-754b5695d5-vg55f"
Oct 30 21:18:48.631466 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:18:48.631440    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d8e0e5b-a716-4dd5-b2d6-8a491fe24065-home\") pod \"simulator-single-main-11b629-kserve-754b5695d5-vg55f\" (UID: \"0d8e0e5b-a716-4dd5-b2d6-8a491fe24065\") " pod="e2e-models-e2e-worker-single-main-11b629/simulator-single-main-11b629-kserve-754b5695d5-vg55f"
Oct 30 21:18:48.631576 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:18:48.631483    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d8e0e5b-a716-4dd5-b2d6-8a491fe24065-model-cache\") pod \"simulator-single-main-11b629-kserve-754b5695d5-vg55f\" (UID: \"0d8e0e5b-a716-4dd5-b2d6-8a491fe24065\") " pod="e2e-models-e2e-worker-single-main-11b629/simulator-single-main-11b629-kserve-754b5695d5-vg55f"
Oct 30 21:18:48.631576 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:18:48.631522    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d8e0e5b-a716-4dd5-b2d6-8a491fe24065-tls-certs\") pod \"simulator-single-main-11b629-kserve-754b5695d5-vg55f\" (UID: \"0d8e0e5b-a716-4dd5-b2d6-8a491fe24065\") " pod="e2e-models-e2e-worker-single-main-11b629/simulator-single-main-11b629-kserve-754b5695d5-vg55f"
Oct 30 21:18:48.631696 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:18:48.631594    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d8e0e5b-a716-4dd5-b2d6-8a491fe24065-tmp-dir\") pod \"simulator-single-main-11b629-kserve-754b5695d5-vg55f\" (UID: \"0d8e0e5b-a716-4dd5-b2d6-8a491fe24065\") " pod="e2e-models-e2e-worker-single-main-11b629/simulator-single-main-11b629-kserve-754b5695d5-vg55f"
Oct 30 21:18:48.631696 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:18:48.631647    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d8e0e5b-a716-4dd5-b2d6-8a491fe24065-dshm\") pod \"simulator-single-main-11b629-kserve-754b5695d5-vg55f\" (UID: \"0d8e0e5b-a716-4dd5-b2d6-8a491fe24065\") " pod="e2e-models-e2e-worker-single-main-11b629/simulator-single-main-11b629-kserve-754b5695d5-vg55f"
Oct 30 21:18:48.732494 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:18:48.732404    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d8e0e5b-a716-4dd5-b2d6-8a491fe24065-home\") pod \"simulator-single-main-11b629-kserve-754b5695d5-vg55f\" (UID: \"0d8e0e5b-a716-4dd5-b2d6-8a491fe24065\") " pod="e2e-models-e2e-worker-single-main-11b629/simulator-single-main-11b629-kserve-754b5695d5-vg55f"
Oct 30 21:18:48.732494 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:18:48.732446    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d8e0e5b-a716-4dd5-b2d6-8a491fe24065-model-cache\") pod \"simulator-single-main-11b629-kserve-754b5695d5-vg55f\" (UID: \"0d8e0e5b-a716-4dd5-b2d6-8a491fe24065\") " pod="e2e-models-e2e-worker-single-main-11b629/simulator-single-main-11b629-kserve-754b5695d5-vg55f"
Oct 30 21:18:48.732494 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:18:48.732477    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d8e0e5b-a716-4dd5-b2d6-8a491fe24065-tls-certs\") pod \"simulator-single-main-11b629-kserve-754b5695d5-vg55f\" (UID: \"0d8e0e5b-a716-4dd5-b2d6-8a491fe24065\") " pod="e2e-models-e2e-worker-single-main-11b629/simulator-single-main-11b629-kserve-754b5695d5-vg55f"
Oct 30 21:18:48.732797 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:18:48.732539    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d8e0e5b-a716-4dd5-b2d6-8a491fe24065-tmp-dir\") pod \"simulator-single-main-11b629-kserve-754b5695d5-vg55f\" (UID: \"0d8e0e5b-a716-4dd5-b2d6-8a491fe24065\") " pod="e2e-models-e2e-worker-single-main-11b629/simulator-single-main-11b629-kserve-754b5695d5-vg55f"
Oct 30 21:18:48.732797 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:18:48.732587    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d8e0e5b-a716-4dd5-b2d6-8a491fe24065-dshm\") pod \"simulator-single-main-11b629-kserve-754b5695d5-vg55f\" (UID: \"0d8e0e5b-a716-4dd5-b2d6-8a491fe24065\") " pod="e2e-models-e2e-worker-single-main-11b629/simulator-single-main-11b629-kserve-754b5695d5-vg55f"
Oct 30 21:18:48.732797 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:18:48.732625    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzn5t\" (UniqueName: \"kubernetes.io/projected/0d8e0e5b-a716-4dd5-b2d6-8a491fe24065-kube-api-access-zzn5t\") pod \"simulator-single-main-11b629-kserve-754b5695d5-vg55f\" (UID: \"0d8e0e5b-a716-4dd5-b2d6-8a491fe24065\") " pod="e2e-models-e2e-worker-single-main-11b629/simulator-single-main-11b629-kserve-754b5695d5-vg55f"
Oct 30 21:18:48.732954 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:18:48.732887    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d8e0e5b-a716-4dd5-b2d6-8a491fe24065-home\") pod \"simulator-single-main-11b629-kserve-754b5695d5-vg55f\" (UID: \"0d8e0e5b-a716-4dd5-b2d6-8a491fe24065\") " pod="e2e-models-e2e-worker-single-main-11b629/simulator-single-main-11b629-kserve-754b5695d5-vg55f"
Oct 30 21:18:48.732954 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:18:48.732931    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d8e0e5b-a716-4dd5-b2d6-8a491fe24065-model-cache\") pod \"simulator-single-main-11b629-kserve-754b5695d5-vg55f\" (UID: \"0d8e0e5b-a716-4dd5-b2d6-8a491fe24065\") " pod="e2e-models-e2e-worker-single-main-11b629/simulator-single-main-11b629-kserve-754b5695d5-vg55f"
Oct 30 21:18:48.733179 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:18:48.733156    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d8e0e5b-a716-4dd5-b2d6-8a491fe24065-tmp-dir\") pod \"simulator-single-main-11b629-kserve-754b5695d5-vg55f\" (UID: \"0d8e0e5b-a716-4dd5-b2d6-8a491fe24065\") " pod="e2e-models-e2e-worker-single-main-11b629/simulator-single-main-11b629-kserve-754b5695d5-vg55f"
Oct 30 21:18:48.735118 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:18:48.735093    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d8e0e5b-a716-4dd5-b2d6-8a491fe24065-dshm\") pod \"simulator-single-main-11b629-kserve-754b5695d5-vg55f\" (UID: \"0d8e0e5b-a716-4dd5-b2d6-8a491fe24065\") " pod="e2e-models-e2e-worker-single-main-11b629/simulator-single-main-11b629-kserve-754b5695d5-vg55f"
Oct 30 21:18:48.735560 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:18:48.735543    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d8e0e5b-a716-4dd5-b2d6-8a491fe24065-tls-certs\") pod \"simulator-single-main-11b629-kserve-754b5695d5-vg55f\" (UID: \"0d8e0e5b-a716-4dd5-b2d6-8a491fe24065\") " pod="e2e-models-e2e-worker-single-main-11b629/simulator-single-main-11b629-kserve-754b5695d5-vg55f"
Oct 30 21:18:48.744881 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:18:48.744853    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzn5t\" (UniqueName: \"kubernetes.io/projected/0d8e0e5b-a716-4dd5-b2d6-8a491fe24065-kube-api-access-zzn5t\") pod \"simulator-single-main-11b629-kserve-754b5695d5-vg55f\" (UID: \"0d8e0e5b-a716-4dd5-b2d6-8a491fe24065\") " pod="e2e-models-e2e-worker-single-main-11b629/simulator-single-main-11b629-kserve-754b5695d5-vg55f"
Oct 30 21:18:48.768578 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:18:48.768537    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-11b629/simulator-single-main-11b629-kserve-754b5695d5-vg55f"
Oct 30 21:18:49.053208 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:18:49.053181    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-11b629/simulator-single-main-11b629-kserve-754b5695d5-vg55f"]
Oct 30 21:18:49.055429 ip-10-0-131-125 kubenswrapper[4521]: W1030 21:18:49.055402    4521 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0d8e0e5b_a716_4dd5_b2d6_8a491fe24065.slice/crio-954195b99af0e2adbf7f215178b81c65668a44ced48d0112df4e9a933a8e285e WatchSource:0}: Error finding container 954195b99af0e2adbf7f215178b81c65668a44ced48d0112df4e9a933a8e285e: Status 404 returned error can't find the container with id 954195b99af0e2adbf7f215178b81c65668a44ced48d0112df4e9a933a8e285e
Oct 30 21:18:49.615049 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:18:49.615013    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-11b629/simulator-single-main-11b629-kserve-754b5695d5-vg55f" event={"ID":"0d8e0e5b-a716-4dd5-b2d6-8a491fe24065","Type":"ContainerStarted","Data":"1bbc37b60ce10873e389790d20e2c2fe67ca89f18100a61d7672351cb90f0c09"}
Oct 30 21:18:49.615487 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:18:49.615056    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-11b629/simulator-single-main-11b629-kserve-754b5695d5-vg55f" event={"ID":"0d8e0e5b-a716-4dd5-b2d6-8a491fe24065","Type":"ContainerStarted","Data":"954195b99af0e2adbf7f215178b81c65668a44ced48d0112df4e9a933a8e285e"}
Oct 30 21:18:49.635528 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:18:49.635466    4521 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-11b629/simulator-single-main-11b629-kserve-754b5695d5-vg55f" podStartSLOduration=1.6354477589999998 podStartE2EDuration="1.635447759s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:18:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:18:49.633679596 +0000 UTC m=+2396.979300835" watchObservedRunningTime="2026-10-30 21:18:49.635447759 +0000 UTC m=+2396.981068999"
Oct 30 21:18:49.769110 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:18:49.769070    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-11b629/simulator-single-main-11b629-kserve-754b5695d5-vg55f"
Oct 30 21:18:50.288531 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:18:50.288490    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4kxb"]
Oct 30 21:18:58.769074 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:18:58.769018    4521 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-11b629/simulator-single-main-11b629-kserve-754b5695d5-vg55f"
Oct 30 21:18:58.781900 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:18:58.781870    4521 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-11b629/simulator-single-main-11b629-kserve-754b5695d5-vg55f"
Oct 30 21:18:59.677746 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:18:59.677718    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-11b629/simulator-single-main-11b629-kserve-754b5695d5-vg55f"
Oct 30 21:19:05.514772 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:19:05.514730    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4kxb"]
Oct 30 21:19:16.427878 ip-10-0-131-125 kubenswrapper[4521]: E1030 21:19:16.427834    4521 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lkn2k" containerName="extractor"
Oct 30 21:19:18.233311 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:19:18.233267    4521 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3dc5db79-r99pk"]
Oct 30 21:19:18.237051 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:19:18.237029    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dc5db79-r99pk"
Oct 30 21:19:18.243566 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:19:18.243537    4521 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:19:18.243740 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:19:18.243705    4521 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c45pw\""
Oct 30 21:19:18.243740 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:19:18.243606    4521 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:19:18.243740 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:19:18.243551    4521 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:19:18.246976 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:19:18.246951    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3dc5db79-r99pk"]
Oct 30 21:19:18.306004 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:19:18.305962    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45lzd\" (UniqueName: \"kubernetes.io/projected/18b7bc42-cacf-43cd-92f0-d600df31689e-kube-api-access-45lzd\") pod \"maas-api-revoke-sub-3dc5db79-r99pk\" (UID: \"18b7bc42-cacf-43cd-92f0-d600df31689e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dc5db79-r99pk"
Oct 30 21:19:18.306194 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:19:18.306020    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18b7bc42-cacf-43cd-92f0-d600df31689e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3dc5db79-r99pk\" (UID: \"18b7bc42-cacf-43cd-92f0-d600df31689e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dc5db79-r99pk"
Oct 30 21:19:18.306194 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:19:18.306089    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18b7bc42-cacf-43cd-92f0-d600df31689e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3dc5db79-r99pk\" (UID: \"18b7bc42-cacf-43cd-92f0-d600df31689e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dc5db79-r99pk"
Oct 30 21:19:18.407525 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:19:18.407489    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45lzd\" (UniqueName: \"kubernetes.io/projected/18b7bc42-cacf-43cd-92f0-d600df31689e-kube-api-access-45lzd\") pod \"maas-api-revoke-sub-3dc5db79-r99pk\" (UID: \"18b7bc42-cacf-43cd-92f0-d600df31689e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dc5db79-r99pk"
Oct 30 21:19:18.407709 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:19:18.407548    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18b7bc42-cacf-43cd-92f0-d600df31689e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3dc5db79-r99pk\" (UID: \"18b7bc42-cacf-43cd-92f0-d600df31689e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dc5db79-r99pk"
Oct 30 21:19:18.407709 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:19:18.407591    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18b7bc42-cacf-43cd-92f0-d600df31689e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3dc5db79-r99pk\" (UID: \"18b7bc42-cacf-43cd-92f0-d600df31689e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dc5db79-r99pk"
Oct 30 21:19:18.408260 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:19:18.408235    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18b7bc42-cacf-43cd-92f0-d600df31689e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3dc5db79-r99pk\" (UID: \"18b7bc42-cacf-43cd-92f0-d600df31689e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dc5db79-r99pk"
Oct 30 21:19:18.409957 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:19:18.409926    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18b7bc42-cacf-43cd-92f0-d600df31689e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3dc5db79-r99pk\" (UID: \"18b7bc42-cacf-43cd-92f0-d600df31689e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dc5db79-r99pk"
Oct 30 21:19:18.420812 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:19:18.420783    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45lzd\" (UniqueName: \"kubernetes.io/projected/18b7bc42-cacf-43cd-92f0-d600df31689e-kube-api-access-45lzd\") pod \"maas-api-revoke-sub-3dc5db79-r99pk\" (UID: \"18b7bc42-cacf-43cd-92f0-d600df31689e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dc5db79-r99pk"
Oct 30 21:19:18.551632 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:19:18.551547    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dc5db79-r99pk"
Oct 30 21:19:18.832454 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:19:18.832421    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3dc5db79-r99pk"]
Oct 30 21:19:18.833662 ip-10-0-131-125 kubenswrapper[4521]: W1030 21:19:18.833636    4521 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18b7bc42_cacf_43cd_92f0_d600df31689e.slice/crio-4ce40a636dd3f14974114fe4d3a8bb34836e3e8921fc4e364677b9bb4040bd0c WatchSource:0}: Error finding container 4ce40a636dd3f14974114fe4d3a8bb34836e3e8921fc4e364677b9bb4040bd0c: Status 404 returned error can't find the container with id 4ce40a636dd3f14974114fe4d3a8bb34836e3e8921fc4e364677b9bb4040bd0c
Oct 30 21:19:19.770122 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:19:19.770080    4521 generic.go:356] "Generic (PLEG): container finished" podID="18b7bc42-cacf-43cd-92f0-d600df31689e" containerID="e19f9a9def550c48fdcf7accf8fda4d13b6ef8ef72945e796c212fe0f6366264" exitCode=0
Oct 30 21:19:19.770616 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:19:19.770161    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dc5db79-r99pk" event={"ID":"18b7bc42-cacf-43cd-92f0-d600df31689e","Type":"ContainerDied","Data":"e19f9a9def550c48fdcf7accf8fda4d13b6ef8ef72945e796c212fe0f6366264"}
Oct 30 21:19:19.770616 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:19:19.770213    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dc5db79-r99pk" event={"ID":"18b7bc42-cacf-43cd-92f0-d600df31689e","Type":"ContainerStarted","Data":"4ce40a636dd3f14974114fe4d3a8bb34836e3e8921fc4e364677b9bb4040bd0c"}
Oct 30 21:19:20.906937 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:19:20.906909    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dc5db79-r99pk"
Oct 30 21:19:21.034195 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:19:21.033114    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18b7bc42-cacf-43cd-92f0-d600df31689e-kube-api-access-45lzd\" (UniqueName: \"kubernetes.io/projected/18b7bc42-cacf-43cd-92f0-d600df31689e-kube-api-access-45lzd\") pod \"18b7bc42-cacf-43cd-92f0-d600df31689e\" (UID: \"18b7bc42-cacf-43cd-92f0-d600df31689e\") "
Oct 30 21:19:21.034195 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:19:21.033173    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18b7bc42-cacf-43cd-92f0-d600df31689e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18b7bc42-cacf-43cd-92f0-d600df31689e-maas-api-openshift-ca\") pod \"18b7bc42-cacf-43cd-92f0-d600df31689e\" (UID: \"18b7bc42-cacf-43cd-92f0-d600df31689e\") "
Oct 30 21:19:21.034195 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:19:21.033211    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18b7bc42-cacf-43cd-92f0-d600df31689e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18b7bc42-cacf-43cd-92f0-d600df31689e-maas-api-xks-ca\") pod \"18b7bc42-cacf-43cd-92f0-d600df31689e\" (UID: \"18b7bc42-cacf-43cd-92f0-d600df31689e\") "
Oct 30 21:19:21.034195 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:19:21.034077    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18b7bc42-cacf-43cd-92f0-d600df31689e-maas-api-openshift-ca" pod "18b7bc42-cacf-43cd-92f0-d600df31689e" (UID: "18b7bc42-cacf-43cd-92f0-d600df31689e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:19:21.038834 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:19:21.038799    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18b7bc42-cacf-43cd-92f0-d600df31689e-kube-api-access-45lzd" pod "18b7bc42-cacf-43cd-92f0-d600df31689e" (UID: "18b7bc42-cacf-43cd-92f0-d600df31689e"). InnerVolumeSpecName "kube-api-access-45lzd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:19:21.039003 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:19:21.038834    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18b7bc42-cacf-43cd-92f0-d600df31689e-maas-api-xks-ca" pod "18b7bc42-cacf-43cd-92f0-d600df31689e" (UID: "18b7bc42-cacf-43cd-92f0-d600df31689e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:19:21.134468 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:19:21.134435    4521 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45lzd\" (UniqueName: \"kubernetes.io/projected/18b7bc42-cacf-43cd-92f0-d600df31689e-kube-api-access-45lzd\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:19:21.134468 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:19:21.134466    4521 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18b7bc42-cacf-43cd-92f0-d600df31689e-maas-api-openshift-ca\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:19:21.134680 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:19:21.134480    4521 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18b7bc42-cacf-43cd-92f0-d600df31689e-maas-api-xks-ca\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:19:21.781693 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:19:21.781656    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dc5db79-r99pk" event={"ID":"18b7bc42-cacf-43cd-92f0-d600df31689e","Type":"ContainerDied","Data":"4ce40a636dd3f14974114fe4d3a8bb34836e3e8921fc4e364677b9bb4040bd0c"}
Oct 30 21:19:21.781693 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:19:21.781693    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dc5db79-r99pk"
Oct 30 21:19:21.781918 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:19:21.781707    4521 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4ce40a636dd3f14974114fe4d3a8bb34836e3e8921fc4e364677b9bb4040bd0c"
Oct 30 21:19:29.156909 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:19:29.156872    4521 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-11b629-q8nj4"]
Oct 30 21:19:29.157298 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:19:29.157263    4521 cpu_manager.go:414] "containerMap: removing container" podUID="18b7bc42-cacf-43cd-92f0-d600df31689e" containerName="revoke-keys"
Oct 30 21:19:29.157298 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:19:29.157274    4521 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18b7bc42-cacf-43cd-92f0-d600df31689e" containerName="revoke-keys"
Oct 30 21:19:29.157403 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:19:29.157356    4521 memory_manager.go:365] "RemoveStaleState removing state" podUID="18b7bc42-cacf-43cd-92f0-d600df31689e" containerName="revoke-keys"
Oct 30 21:19:29.160442 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:19:29.160418    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-11b629-q8nj4"
Oct 30 21:19:29.169010 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:19:29.168989    4521 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:19:29.169181 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:19:29.168994    4521 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:19:29.169265 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:19:29.168994    4521 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c45pw\""
Oct 30 21:19:29.169265 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:19:29.169050    4521 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:19:29.171186 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:19:29.171164    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-11b629-q8nj4"]
Oct 30 21:19:29.317913 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:19:29.317867    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcd44a7e-038d-4570-b940-d4072a85cd3d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-11b629-q8nj4\" (UID: \"fcd44a7e-038d-4570-b940-d4072a85cd3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-11b629-q8nj4"
Oct 30 21:19:29.318106 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:19:29.318016    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcd44a7e-038d-4570-b940-d4072a85cd3d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-11b629-q8nj4\" (UID: \"fcd44a7e-038d-4570-b940-d4072a85cd3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-11b629-q8nj4"
Oct 30 21:19:29.318106 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:19:29.318054    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbc44\" (UniqueName: \"kubernetes.io/projected/fcd44a7e-038d-4570-b940-d4072a85cd3d-kube-api-access-qbc44\") pod \"maas-api-revoke-keys-e2e-worker-single-main-11b629-q8nj4\" (UID: \"fcd44a7e-038d-4570-b940-d4072a85cd3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-11b629-q8nj4"
Oct 30 21:19:29.385145 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:19:29.385110    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4kxb"]
Oct 30 21:19:29.418808 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:19:29.418733    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcd44a7e-038d-4570-b940-d4072a85cd3d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-11b629-q8nj4\" (UID: \"fcd44a7e-038d-4570-b940-d4072a85cd3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-11b629-q8nj4"
Oct 30 21:19:29.418808 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:19:29.418785    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcd44a7e-038d-4570-b940-d4072a85cd3d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-11b629-q8nj4\" (UID: \"fcd44a7e-038d-4570-b940-d4072a85cd3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-11b629-q8nj4"
Oct 30 21:19:29.419002 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:19:29.418813    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbc44\" (UniqueName: \"kubernetes.io/projected/fcd44a7e-038d-4570-b940-d4072a85cd3d-kube-api-access-qbc44\") pod \"maas-api-revoke-keys-e2e-worker-single-main-11b629-q8nj4\" (UID: \"fcd44a7e-038d-4570-b940-d4072a85cd3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-11b629-q8nj4"
Oct 30 21:19:29.419599 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:19:29.419573    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcd44a7e-038d-4570-b940-d4072a85cd3d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-11b629-q8nj4\" (UID: \"fcd44a7e-038d-4570-b940-d4072a85cd3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-11b629-q8nj4"
Oct 30 21:19:29.421318 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:19:29.421291    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcd44a7e-038d-4570-b940-d4072a85cd3d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-11b629-q8nj4\" (UID: \"fcd44a7e-038d-4570-b940-d4072a85cd3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-11b629-q8nj4"
Oct 30 21:19:29.432173 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:19:29.432148    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbc44\" (UniqueName: \"kubernetes.io/projected/fcd44a7e-038d-4570-b940-d4072a85cd3d-kube-api-access-qbc44\") pod \"maas-api-revoke-keys-e2e-worker-single-main-11b629-q8nj4\" (UID: \"fcd44a7e-038d-4570-b940-d4072a85cd3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-11b629-q8nj4"
Oct 30 21:19:29.474382 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:19:29.474348    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-11b629-q8nj4"
Oct 30 21:19:29.752019 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:19:29.751990    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-11b629-q8nj4"]
Oct 30 21:19:29.753762 ip-10-0-131-125 kubenswrapper[4521]: W1030 21:19:29.753734    4521 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcd44a7e_038d_4570_b940_d4072a85cd3d.slice/crio-73ab65e595026d75fe2b3c675e09c9ca24c468a78931bbcf1e4ca7f4188d6e69 WatchSource:0}: Error finding container 73ab65e595026d75fe2b3c675e09c9ca24c468a78931bbcf1e4ca7f4188d6e69: Status 404 returned error can't find the container with id 73ab65e595026d75fe2b3c675e09c9ca24c468a78931bbcf1e4ca7f4188d6e69
Oct 30 21:19:29.825078 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:19:29.825020    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-11b629-q8nj4" event={"ID":"fcd44a7e-038d-4570-b940-d4072a85cd3d","Type":"ContainerStarted","Data":"73ab65e595026d75fe2b3c675e09c9ca24c468a78931bbcf1e4ca7f4188d6e69"}
Oct 30 21:19:30.831483 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:19:30.831447    4521 generic.go:356] "Generic (PLEG): container finished" podID="fcd44a7e-038d-4570-b940-d4072a85cd3d" containerID="c8e831d9fa0b8730ff0fbdfde3da0920a95db2c629cb231e481ec2fbfe25679d" exitCode=0
Oct 30 21:19:30.831874 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:19:30.831496    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-11b629-q8nj4" event={"ID":"fcd44a7e-038d-4570-b940-d4072a85cd3d","Type":"ContainerDied","Data":"c8e831d9fa0b8730ff0fbdfde3da0920a95db2c629cb231e481ec2fbfe25679d"}
Oct 30 21:19:31.972085 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:19:31.972060    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-11b629-q8nj4"
Oct 30 21:19:32.143979 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:19:32.143889    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fcd44a7e-038d-4570-b940-d4072a85cd3d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcd44a7e-038d-4570-b940-d4072a85cd3d-maas-api-openshift-ca\") pod \"fcd44a7e-038d-4570-b940-d4072a85cd3d\" (UID: \"fcd44a7e-038d-4570-b940-d4072a85cd3d\") "
Oct 30 21:19:32.143979 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:19:32.143956    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcd44a7e-038d-4570-b940-d4072a85cd3d-kube-api-access-qbc44\" (UniqueName: \"kubernetes.io/projected/fcd44a7e-038d-4570-b940-d4072a85cd3d-kube-api-access-qbc44\") pod \"fcd44a7e-038d-4570-b940-d4072a85cd3d\" (UID: \"fcd44a7e-038d-4570-b940-d4072a85cd3d\") "
Oct 30 21:19:32.144213 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:19:32.144010    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fcd44a7e-038d-4570-b940-d4072a85cd3d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcd44a7e-038d-4570-b940-d4072a85cd3d-maas-api-xks-ca\") pod \"fcd44a7e-038d-4570-b940-d4072a85cd3d\" (UID: \"fcd44a7e-038d-4570-b940-d4072a85cd3d\") "
Oct 30 21:19:32.144277 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:19:32.144257    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcd44a7e-038d-4570-b940-d4072a85cd3d-maas-api-openshift-ca" pod "fcd44a7e-038d-4570-b940-d4072a85cd3d" (UID: "fcd44a7e-038d-4570-b940-d4072a85cd3d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:19:32.144388 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:19:32.144372    4521 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcd44a7e-038d-4570-b940-d4072a85cd3d-maas-api-openshift-ca\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:19:32.146195 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:19:32.146173    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcd44a7e-038d-4570-b940-d4072a85cd3d-maas-api-xks-ca" pod "fcd44a7e-038d-4570-b940-d4072a85cd3d" (UID: "fcd44a7e-038d-4570-b940-d4072a85cd3d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:19:32.146272 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:19:32.146216    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcd44a7e-038d-4570-b940-d4072a85cd3d-kube-api-access-qbc44" pod "fcd44a7e-038d-4570-b940-d4072a85cd3d" (UID: "fcd44a7e-038d-4570-b940-d4072a85cd3d"). InnerVolumeSpecName "kube-api-access-qbc44". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:19:32.245585 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:19:32.245552    4521 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbc44\" (UniqueName: \"kubernetes.io/projected/fcd44a7e-038d-4570-b940-d4072a85cd3d-kube-api-access-qbc44\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:19:32.245585 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:19:32.245582    4521 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcd44a7e-038d-4570-b940-d4072a85cd3d-maas-api-xks-ca\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:19:32.843769 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:19:32.843731    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-11b629-q8nj4" event={"ID":"fcd44a7e-038d-4570-b940-d4072a85cd3d","Type":"ContainerDied","Data":"73ab65e595026d75fe2b3c675e09c9ca24c468a78931bbcf1e4ca7f4188d6e69"}
Oct 30 21:19:32.843969 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:19:32.843790    4521 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="73ab65e595026d75fe2b3c675e09c9ca24c468a78931bbcf1e4ca7f4188d6e69"
Oct 30 21:19:32.843969 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:19:32.843743    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-11b629-q8nj4"
Oct 30 21:19:33.948911 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:19:33.948877    4521 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-11b629/simulator-single-main-11b629-kserve-754b5695d5-vg55f"]
Oct 30 21:19:39.894217 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:19:39.894179    4521 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-11b629-q8nj4"]
Oct 30 21:19:39.896734 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:19:39.896713    4521 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-11b629-q8nj4"]
Oct 30 21:19:40.791386 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:19:40.791348    4521 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fcd44a7e-038d-4570-b940-d4072a85cd3d" path="/var/lib/kubelet/pods/fcd44a7e-038d-4570-b940-d4072a85cd3d/volumes"
Oct 30 21:19:41.315536 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:19:41.315500    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4kxb"]
Oct 30 21:19:46.429140 ip-10-0-131-125 kubenswrapper[4521]: E1030 21:19:46.429102    4521 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lkn2k" containerName="extractor"
Oct 30 21:19:48.968378 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:19:48.968335    4521 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-11b629/simulator-single-main-11b629-kserve-754b5695d5-vg55f" podUID="0d8e0e5b-a716-4dd5-b2d6-8a491fe24065" containerName="main" containerID="cri-o://1bbc37b60ce10873e389790d20e2c2fe67ca89f18100a61d7672351cb90f0c09" gracePeriod=45
Oct 30 21:19:49.213682 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:19:49.213655    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-11b629/simulator-single-main-11b629-kserve-754b5695d5-vg55f"
Oct 30 21:19:49.298235 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:19:49.298148    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d8e0e5b-a716-4dd5-b2d6-8a491fe24065-home\" (UniqueName: \"kubernetes.io/empty-dir/0d8e0e5b-a716-4dd5-b2d6-8a491fe24065-home\") pod \"0d8e0e5b-a716-4dd5-b2d6-8a491fe24065\" (UID: \"0d8e0e5b-a716-4dd5-b2d6-8a491fe24065\") "
Oct 30 21:19:49.298431 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:19:49.298248    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d8e0e5b-a716-4dd5-b2d6-8a491fe24065-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d8e0e5b-a716-4dd5-b2d6-8a491fe24065-tmp-dir\") pod \"0d8e0e5b-a716-4dd5-b2d6-8a491fe24065\" (UID: \"0d8e0e5b-a716-4dd5-b2d6-8a491fe24065\") "
Oct 30 21:19:49.298431 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:19:49.298307    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d8e0e5b-a716-4dd5-b2d6-8a491fe24065-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d8e0e5b-a716-4dd5-b2d6-8a491fe24065-model-cache\") pod \"0d8e0e5b-a716-4dd5-b2d6-8a491fe24065\" (UID: \"0d8e0e5b-a716-4dd5-b2d6-8a491fe24065\") "
Oct 30 21:19:49.298431 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:19:49.298355    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d8e0e5b-a716-4dd5-b2d6-8a491fe24065-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d8e0e5b-a716-4dd5-b2d6-8a491fe24065-dshm\") pod \"0d8e0e5b-a716-4dd5-b2d6-8a491fe24065\" (UID: \"0d8e0e5b-a716-4dd5-b2d6-8a491fe24065\") "
Oct 30 21:19:49.298431 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:19:49.298386    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d8e0e5b-a716-4dd5-b2d6-8a491fe24065-tls-certs\" (UniqueName: \"kubernetes.io/secret/0d8e0e5b-a716-4dd5-b2d6-8a491fe24065-tls-certs\") pod \"0d8e0e5b-a716-4dd5-b2d6-8a491fe24065\" (UID: \"0d8e0e5b-a716-4dd5-b2d6-8a491fe24065\") "
Oct 30 21:19:49.298661 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:19:49.298447    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d8e0e5b-a716-4dd5-b2d6-8a491fe24065-kube-api-access-zzn5t\" (UniqueName: \"kubernetes.io/projected/0d8e0e5b-a716-4dd5-b2d6-8a491fe24065-kube-api-access-zzn5t\") pod \"0d8e0e5b-a716-4dd5-b2d6-8a491fe24065\" (UID: \"0d8e0e5b-a716-4dd5-b2d6-8a491fe24065\") "
Oct 30 21:19:49.298661 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:19:49.298502    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d8e0e5b-a716-4dd5-b2d6-8a491fe24065-home" pod "0d8e0e5b-a716-4dd5-b2d6-8a491fe24065" (UID: "0d8e0e5b-a716-4dd5-b2d6-8a491fe24065"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:19:49.298661 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:19:49.298512    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d8e0e5b-a716-4dd5-b2d6-8a491fe24065-tmp-dir" pod "0d8e0e5b-a716-4dd5-b2d6-8a491fe24065" (UID: "0d8e0e5b-a716-4dd5-b2d6-8a491fe24065"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:19:49.298661 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:19:49.298595    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d8e0e5b-a716-4dd5-b2d6-8a491fe24065-model-cache" pod "0d8e0e5b-a716-4dd5-b2d6-8a491fe24065" (UID: "0d8e0e5b-a716-4dd5-b2d6-8a491fe24065"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:19:49.298865 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:19:49.298793    4521 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d8e0e5b-a716-4dd5-b2d6-8a491fe24065-tmp-dir\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:19:49.298865 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:19:49.298810    4521 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d8e0e5b-a716-4dd5-b2d6-8a491fe24065-model-cache\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:19:49.298865 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:19:49.298820    4521 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d8e0e5b-a716-4dd5-b2d6-8a491fe24065-home\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:19:49.300698 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:19:49.300671    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d8e0e5b-a716-4dd5-b2d6-8a491fe24065-kube-api-access-zzn5t" pod "0d8e0e5b-a716-4dd5-b2d6-8a491fe24065" (UID: "0d8e0e5b-a716-4dd5-b2d6-8a491fe24065"). InnerVolumeSpecName "kube-api-access-zzn5t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:19:49.300934 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:19:49.300899    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d8e0e5b-a716-4dd5-b2d6-8a491fe24065-tls-certs" pod "0d8e0e5b-a716-4dd5-b2d6-8a491fe24065" (UID: "0d8e0e5b-a716-4dd5-b2d6-8a491fe24065"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:19:49.300934 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:19:49.300915    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d8e0e5b-a716-4dd5-b2d6-8a491fe24065-dshm" pod "0d8e0e5b-a716-4dd5-b2d6-8a491fe24065" (UID: "0d8e0e5b-a716-4dd5-b2d6-8a491fe24065"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:19:49.399443 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:19:49.399411    4521 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d8e0e5b-a716-4dd5-b2d6-8a491fe24065-dshm\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:19:49.399443 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:19:49.399439    4521 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d8e0e5b-a716-4dd5-b2d6-8a491fe24065-tls-certs\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:19:49.399443 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:19:49.399448    4521 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zzn5t\" (UniqueName: \"kubernetes.io/projected/0d8e0e5b-a716-4dd5-b2d6-8a491fe24065-kube-api-access-zzn5t\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:19:49.937644 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:19:49.937605    4521 generic.go:356] "Generic (PLEG): container finished" podID="0d8e0e5b-a716-4dd5-b2d6-8a491fe24065" containerID="1bbc37b60ce10873e389790d20e2c2fe67ca89f18100a61d7672351cb90f0c09" exitCode=0
Oct 30 21:19:49.937829 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:19:49.937657    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-11b629/simulator-single-main-11b629-kserve-754b5695d5-vg55f"
Oct 30 21:19:49.937829 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:19:49.937663    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-11b629/simulator-single-main-11b629-kserve-754b5695d5-vg55f" event={"ID":"0d8e0e5b-a716-4dd5-b2d6-8a491fe24065","Type":"ContainerDied","Data":"1bbc37b60ce10873e389790d20e2c2fe67ca89f18100a61d7672351cb90f0c09"}
Oct 30 21:19:49.937829 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:19:49.937703    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-11b629/simulator-single-main-11b629-kserve-754b5695d5-vg55f" event={"ID":"0d8e0e5b-a716-4dd5-b2d6-8a491fe24065","Type":"ContainerDied","Data":"954195b99af0e2adbf7f215178b81c65668a44ced48d0112df4e9a933a8e285e"}
Oct 30 21:19:49.937829 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:19:49.937725    4521 scope.go:122] "RemoveContainer" containerID="1bbc37b60ce10873e389790d20e2c2fe67ca89f18100a61d7672351cb90f0c09"
Oct 30 21:19:49.947186 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:19:49.947165    4521 scope.go:122] "RemoveContainer" containerID="1bbc37b60ce10873e389790d20e2c2fe67ca89f18100a61d7672351cb90f0c09"
Oct 30 21:19:49.947486 ip-10-0-131-125 kubenswrapper[4521]: E1030 21:19:49.947460    4521 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bbc37b60ce10873e389790d20e2c2fe67ca89f18100a61d7672351cb90f0c09\": container with ID starting with 1bbc37b60ce10873e389790d20e2c2fe67ca89f18100a61d7672351cb90f0c09 not found: ID does not exist" containerID="1bbc37b60ce10873e389790d20e2c2fe67ca89f18100a61d7672351cb90f0c09"
Oct 30 21:19:49.947566 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:19:49.947495    4521 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bbc37b60ce10873e389790d20e2c2fe67ca89f18100a61d7672351cb90f0c09"} err="failed to get container status \"1bbc37b60ce10873e389790d20e2c2fe67ca89f18100a61d7672351cb90f0c09\": rpc error: code = NotFound desc = could not find container \"1bbc37b60ce10873e389790d20e2c2fe67ca89f18100a61d7672351cb90f0c09\": container with ID starting with 1bbc37b60ce10873e389790d20e2c2fe67ca89f18100a61d7672351cb90f0c09 not found: ID does not exist"
Oct 30 21:19:49.964875 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:19:49.964845    4521 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-11b629/simulator-single-main-11b629-kserve-754b5695d5-vg55f"]
Oct 30 21:19:49.969079 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:19:49.969058    4521 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-11b629/simulator-single-main-11b629-kserve-754b5695d5-vg55f"]
Oct 30 21:19:50.791395 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:19:50.791361    4521 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d8e0e5b-a716-4dd5-b2d6-8a491fe24065" path="/var/lib/kubelet/pods/0d8e0e5b-a716-4dd5-b2d6-8a491fe24065/volumes"
Oct 30 21:19:52.103226 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:19:52.103176    4521 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-3f8108d8-kserve-65df7899df-4wpdz"]
Oct 30 21:19:52.103900 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:19:52.103872    4521 cpu_manager.go:414] "containerMap: removing container" podUID="fcd44a7e-038d-4570-b940-d4072a85cd3d" containerName="revoke-keys"
Oct 30 21:19:52.103941 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:19:52.103904    4521 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcd44a7e-038d-4570-b940-d4072a85cd3d" containerName="revoke-keys"
Oct 30 21:19:52.103976 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:19:52.103946    4521 cpu_manager.go:414] "containerMap: removing container" podUID="0d8e0e5b-a716-4dd5-b2d6-8a491fe24065" containerName="main"
Oct 30 21:19:52.103976 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:19:52.103955    4521 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d8e0e5b-a716-4dd5-b2d6-8a491fe24065" containerName="main"
Oct 30 21:19:52.104068 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:19:52.104055    4521 memory_manager.go:365] "RemoveStaleState removing state" podUID="fcd44a7e-038d-4570-b940-d4072a85cd3d" containerName="revoke-keys"
Oct 30 21:19:52.104104 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:19:52.104078    4521 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d8e0e5b-a716-4dd5-b2d6-8a491fe24065" containerName="main"
Oct 30 21:19:52.107736 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:19:52.107714    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-3f8108d8-kserve-65df7899df-4wpdz"
Oct 30 21:19:52.114519 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:19:52.114494    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-3f8108d8-kserve-65df7899df-4wpdz"]
Oct 30 21:19:52.118780 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:19:52.118756    4521 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-75544\""
Oct 30 21:19:52.120257 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:19:52.120227    4521 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Oct 30 21:19:52.120257 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:19:52.120227    4521 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Oct 30 21:19:52.120519 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:19:52.120278    4521 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-3f8108d8-kserve-self-signed-certs\""
Oct 30 21:19:52.123701 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:19:52.123652    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc64803f-bae9-4ebd-b34e-5667b9eac74a-home\") pod \"e2e-same-id-a-3f8108d8-kserve-65df7899df-4wpdz\" (UID: \"cc64803f-bae9-4ebd-b34e-5667b9eac74a\") " pod="llm/e2e-same-id-a-3f8108d8-kserve-65df7899df-4wpdz"
Oct 30 21:19:52.123834 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:19:52.123811    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjprd\" (UniqueName: \"kubernetes.io/projected/cc64803f-bae9-4ebd-b34e-5667b9eac74a-kube-api-access-wjprd\") pod \"e2e-same-id-a-3f8108d8-kserve-65df7899df-4wpdz\" (UID: \"cc64803f-bae9-4ebd-b34e-5667b9eac74a\") " pod="llm/e2e-same-id-a-3f8108d8-kserve-65df7899df-4wpdz"
Oct 30 21:19:52.123900 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:19:52.123857    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc64803f-bae9-4ebd-b34e-5667b9eac74a-tmp-dir\") pod \"e2e-same-id-a-3f8108d8-kserve-65df7899df-4wpdz\" (UID: \"cc64803f-bae9-4ebd-b34e-5667b9eac74a\") " pod="llm/e2e-same-id-a-3f8108d8-kserve-65df7899df-4wpdz"
Oct 30 21:19:52.123900 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:19:52.123881    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc64803f-bae9-4ebd-b34e-5667b9eac74a-tls-certs\") pod \"e2e-same-id-a-3f8108d8-kserve-65df7899df-4wpdz\" (UID: \"cc64803f-bae9-4ebd-b34e-5667b9eac74a\") " pod="llm/e2e-same-id-a-3f8108d8-kserve-65df7899df-4wpdz"
Oct 30 21:19:52.124011 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:19:52.123974    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc64803f-bae9-4ebd-b34e-5667b9eac74a-dshm\") pod \"e2e-same-id-a-3f8108d8-kserve-65df7899df-4wpdz\" (UID: \"cc64803f-bae9-4ebd-b34e-5667b9eac74a\") " pod="llm/e2e-same-id-a-3f8108d8-kserve-65df7899df-4wpdz"
Oct 30 21:19:52.124072 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:19:52.124031    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc64803f-bae9-4ebd-b34e-5667b9eac74a-model-cache\") pod \"e2e-same-id-a-3f8108d8-kserve-65df7899df-4wpdz\" (UID: \"cc64803f-bae9-4ebd-b34e-5667b9eac74a\") " pod="llm/e2e-same-id-a-3f8108d8-kserve-65df7899df-4wpdz"
Oct 30 21:19:52.224753 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:19:52.224716    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc64803f-bae9-4ebd-b34e-5667b9eac74a-home\") pod \"e2e-same-id-a-3f8108d8-kserve-65df7899df-4wpdz\" (UID: \"cc64803f-bae9-4ebd-b34e-5667b9eac74a\") " pod="llm/e2e-same-id-a-3f8108d8-kserve-65df7899df-4wpdz"
Oct 30 21:19:52.224958 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:19:52.224788    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjprd\" (UniqueName: \"kubernetes.io/projected/cc64803f-bae9-4ebd-b34e-5667b9eac74a-kube-api-access-wjprd\") pod \"e2e-same-id-a-3f8108d8-kserve-65df7899df-4wpdz\" (UID: \"cc64803f-bae9-4ebd-b34e-5667b9eac74a\") " pod="llm/e2e-same-id-a-3f8108d8-kserve-65df7899df-4wpdz"
Oct 30 21:19:52.224958 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:19:52.224820    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc64803f-bae9-4ebd-b34e-5667b9eac74a-tmp-dir\") pod \"e2e-same-id-a-3f8108d8-kserve-65df7899df-4wpdz\" (UID: \"cc64803f-bae9-4ebd-b34e-5667b9eac74a\") " pod="llm/e2e-same-id-a-3f8108d8-kserve-65df7899df-4wpdz"
Oct 30 21:19:52.224958 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:19:52.224840    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc64803f-bae9-4ebd-b34e-5667b9eac74a-tls-certs\") pod \"e2e-same-id-a-3f8108d8-kserve-65df7899df-4wpdz\" (UID: \"cc64803f-bae9-4ebd-b34e-5667b9eac74a\") " pod="llm/e2e-same-id-a-3f8108d8-kserve-65df7899df-4wpdz"
Oct 30 21:19:52.224958 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:19:52.224890    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc64803f-bae9-4ebd-b34e-5667b9eac74a-dshm\") pod \"e2e-same-id-a-3f8108d8-kserve-65df7899df-4wpdz\" (UID: \"cc64803f-bae9-4ebd-b34e-5667b9eac74a\") " pod="llm/e2e-same-id-a-3f8108d8-kserve-65df7899df-4wpdz"
Oct 30 21:19:52.224958 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:19:52.224950    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc64803f-bae9-4ebd-b34e-5667b9eac74a-model-cache\") pod \"e2e-same-id-a-3f8108d8-kserve-65df7899df-4wpdz\" (UID: \"cc64803f-bae9-4ebd-b34e-5667b9eac74a\") " pod="llm/e2e-same-id-a-3f8108d8-kserve-65df7899df-4wpdz"
Oct 30 21:19:52.225269 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:19:52.225202    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc64803f-bae9-4ebd-b34e-5667b9eac74a-home\") pod \"e2e-same-id-a-3f8108d8-kserve-65df7899df-4wpdz\" (UID: \"cc64803f-bae9-4ebd-b34e-5667b9eac74a\") " pod="llm/e2e-same-id-a-3f8108d8-kserve-65df7899df-4wpdz"
Oct 30 21:19:52.225409 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:19:52.225381    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc64803f-bae9-4ebd-b34e-5667b9eac74a-tmp-dir\") pod \"e2e-same-id-a-3f8108d8-kserve-65df7899df-4wpdz\" (UID: \"cc64803f-bae9-4ebd-b34e-5667b9eac74a\") " pod="llm/e2e-same-id-a-3f8108d8-kserve-65df7899df-4wpdz"
Oct 30 21:19:52.225458 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:19:52.225435    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc64803f-bae9-4ebd-b34e-5667b9eac74a-model-cache\") pod \"e2e-same-id-a-3f8108d8-kserve-65df7899df-4wpdz\" (UID: \"cc64803f-bae9-4ebd-b34e-5667b9eac74a\") " pod="llm/e2e-same-id-a-3f8108d8-kserve-65df7899df-4wpdz"
Oct 30 21:19:52.227216 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:19:52.227188    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc64803f-bae9-4ebd-b34e-5667b9eac74a-dshm\") pod \"e2e-same-id-a-3f8108d8-kserve-65df7899df-4wpdz\" (UID: \"cc64803f-bae9-4ebd-b34e-5667b9eac74a\") " pod="llm/e2e-same-id-a-3f8108d8-kserve-65df7899df-4wpdz"
Oct 30 21:19:52.227489 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:19:52.227472    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc64803f-bae9-4ebd-b34e-5667b9eac74a-tls-certs\") pod \"e2e-same-id-a-3f8108d8-kserve-65df7899df-4wpdz\" (UID: \"cc64803f-bae9-4ebd-b34e-5667b9eac74a\") " pod="llm/e2e-same-id-a-3f8108d8-kserve-65df7899df-4wpdz"
Oct 30 21:19:52.248346 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:19:52.245342    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjprd\" (UniqueName: \"kubernetes.io/projected/cc64803f-bae9-4ebd-b34e-5667b9eac74a-kube-api-access-wjprd\") pod \"e2e-same-id-a-3f8108d8-kserve-65df7899df-4wpdz\" (UID: \"cc64803f-bae9-4ebd-b34e-5667b9eac74a\") " pod="llm/e2e-same-id-a-3f8108d8-kserve-65df7899df-4wpdz"
Oct 30 21:19:52.424758 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:19:52.424670    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-3f8108d8-kserve-65df7899df-4wpdz"
Oct 30 21:19:52.705622 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:19:52.705592    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-3f8108d8-kserve-65df7899df-4wpdz"]
Oct 30 21:19:52.707055 ip-10-0-131-125 kubenswrapper[4521]: W1030 21:19:52.707025    4521 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcc64803f_bae9_4ebd_b34e_5667b9eac74a.slice/crio-db129113cc9197c483951cce43850a237811dadd1ded778f31c22e3d6716c637 WatchSource:0}: Error finding container db129113cc9197c483951cce43850a237811dadd1ded778f31c22e3d6716c637: Status 404 returned error can't find the container with id db129113cc9197c483951cce43850a237811dadd1ded778f31c22e3d6716c637
Oct 30 21:19:52.958764 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:19:52.958676    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-3f8108d8-kserve-65df7899df-4wpdz" event={"ID":"cc64803f-bae9-4ebd-b34e-5667b9eac74a","Type":"ContainerStarted","Data":"e856922b5c65bff0d88aa35445822fa40136c150d118c9be06b09de812fbc2fa"}
Oct 30 21:19:52.958764 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:19:52.958726    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-3f8108d8-kserve-65df7899df-4wpdz" event={"ID":"cc64803f-bae9-4ebd-b34e-5667b9eac74a","Type":"ContainerStarted","Data":"db129113cc9197c483951cce43850a237811dadd1ded778f31c22e3d6716c637"}
Oct 30 21:19:52.978378 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:19:52.978299    4521 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-a-3f8108d8-kserve-65df7899df-4wpdz" podStartSLOduration=0.978282716 podStartE2EDuration="978.282716ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:19:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:19:52.977533653 +0000 UTC m=+2460.323154892" watchObservedRunningTime="2026-10-30 21:19:52.978282716 +0000 UTC m=+2460.323903956"
Oct 30 21:19:53.425851 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:19:53.425762    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-3f8108d8-kserve-65df7899df-4wpdz"
Oct 30 21:19:53.822645 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:19:53.822599    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4kxb"]
Oct 30 21:20:02.425280 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:20:02.425235    4521 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-3f8108d8-kserve-65df7899df-4wpdz"
Oct 30 21:20:02.438463 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:20:02.438430    4521 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-3f8108d8-kserve-65df7899df-4wpdz"
Oct 30 21:20:03.038180 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:20:03.038147    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-3f8108d8-kserve-65df7899df-4wpdz"
Oct 30 21:20:09.925135 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:20:09.925043    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4kxb"]
Oct 30 21:20:16.428068 ip-10-0-131-125 kubenswrapper[4521]: E1030 21:20:16.428033    4521 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lkn2k" containerName="extractor"
Oct 30 21:20:22.023930 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:20:22.023891    4521 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7396a035-p69ls"]
Oct 30 21:20:22.026936 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:20:22.026901    4521 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7396a035-p69ls"]
Oct 30 21:20:22.792168 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:20:22.792130    4521 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e261a5da-939e-45a8-a988-f520f446490b" path="/var/lib/kubelet/pods/e261a5da-939e-45a8-a988-f520f446490b/volumes"
Oct 30 21:20:25.915598 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:20:25.915551    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4kxb"]
Oct 30 21:20:36.294236 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:20:36.294198    4521 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-3f8108d8-kserve-65df7899df-4wpdz"]
Oct 30 21:20:37.846735 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:20:37.846695    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4kxb"]
Oct 30 21:20:40.693583 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:20:40.693479    4521 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-mgx2j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:20:40.693583 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:20:40.693569    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-mgx2j" podUID="3f51564c-e79a-437a-9e62-3b2750e0916a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:20:40.737484 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:20:40.737446    4521 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-fw7lj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:20:40.737827 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:20:40.737794    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-fw7lj" podUID="5311e7d9-ac35-419c-a7d8-2bceba0f4f44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:20:43.631345 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:20:43.630452    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4kxb"]
Oct 30 21:20:46.432084 ip-10-0-131-125 kubenswrapper[4521]: E1030 21:20:46.432033    4521 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lkn2k" containerName="extractor"
Oct 30 21:20:51.314812 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:20:51.314745    4521 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-a-3f8108d8-kserve-65df7899df-4wpdz" podUID="cc64803f-bae9-4ebd-b34e-5667b9eac74a" containerName="main" containerID="cri-o://e856922b5c65bff0d88aa35445822fa40136c150d118c9be06b09de812fbc2fa" gracePeriod=45
Oct 30 21:20:51.569843 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:20:51.569782    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-3f8108d8-kserve-65df7899df-4wpdz"
Oct 30 21:20:51.657144 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:20:51.657107    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc64803f-bae9-4ebd-b34e-5667b9eac74a-home\" (UniqueName: \"kubernetes.io/empty-dir/cc64803f-bae9-4ebd-b34e-5667b9eac74a-home\") pod \"cc64803f-bae9-4ebd-b34e-5667b9eac74a\" (UID: \"cc64803f-bae9-4ebd-b34e-5667b9eac74a\") "
Oct 30 21:20:51.657351 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:20:51.657164    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc64803f-bae9-4ebd-b34e-5667b9eac74a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc64803f-bae9-4ebd-b34e-5667b9eac74a-model-cache\") pod \"cc64803f-bae9-4ebd-b34e-5667b9eac74a\" (UID: \"cc64803f-bae9-4ebd-b34e-5667b9eac74a\") "
Oct 30 21:20:51.657351 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:20:51.657204    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc64803f-bae9-4ebd-b34e-5667b9eac74a-kube-api-access-wjprd\" (UniqueName: \"kubernetes.io/projected/cc64803f-bae9-4ebd-b34e-5667b9eac74a-kube-api-access-wjprd\") pod \"cc64803f-bae9-4ebd-b34e-5667b9eac74a\" (UID: \"cc64803f-bae9-4ebd-b34e-5667b9eac74a\") "
Oct 30 21:20:51.657351 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:20:51.657309    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc64803f-bae9-4ebd-b34e-5667b9eac74a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc64803f-bae9-4ebd-b34e-5667b9eac74a-dshm\") pod \"cc64803f-bae9-4ebd-b34e-5667b9eac74a\" (UID: \"cc64803f-bae9-4ebd-b34e-5667b9eac74a\") "
Oct 30 21:20:51.657526 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:20:51.657404    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc64803f-bae9-4ebd-b34e-5667b9eac74a-tls-certs\" (UniqueName: \"kubernetes.io/secret/cc64803f-bae9-4ebd-b34e-5667b9eac74a-tls-certs\") pod \"cc64803f-bae9-4ebd-b34e-5667b9eac74a\" (UID: \"cc64803f-bae9-4ebd-b34e-5667b9eac74a\") "
Oct 30 21:20:51.657526 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:20:51.657462    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc64803f-bae9-4ebd-b34e-5667b9eac74a-home" pod "cc64803f-bae9-4ebd-b34e-5667b9eac74a" (UID: "cc64803f-bae9-4ebd-b34e-5667b9eac74a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:20:51.657526 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:20:51.657471    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc64803f-bae9-4ebd-b34e-5667b9eac74a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc64803f-bae9-4ebd-b34e-5667b9eac74a-tmp-dir\") pod \"cc64803f-bae9-4ebd-b34e-5667b9eac74a\" (UID: \"cc64803f-bae9-4ebd-b34e-5667b9eac74a\") "
Oct 30 21:20:51.657526 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:20:51.657512    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc64803f-bae9-4ebd-b34e-5667b9eac74a-model-cache" pod "cc64803f-bae9-4ebd-b34e-5667b9eac74a" (UID: "cc64803f-bae9-4ebd-b34e-5667b9eac74a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:20:51.657754 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:20:51.657684    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc64803f-bae9-4ebd-b34e-5667b9eac74a-tmp-dir" pod "cc64803f-bae9-4ebd-b34e-5667b9eac74a" (UID: "cc64803f-bae9-4ebd-b34e-5667b9eac74a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:20:51.657853 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:20:51.657833    4521 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc64803f-bae9-4ebd-b34e-5667b9eac74a-tmp-dir\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:20:51.657908 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:20:51.657856    4521 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc64803f-bae9-4ebd-b34e-5667b9eac74a-home\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:20:51.657908 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:20:51.657867    4521 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc64803f-bae9-4ebd-b34e-5667b9eac74a-model-cache\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:20:51.659720 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:20:51.659692    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc64803f-bae9-4ebd-b34e-5667b9eac74a-kube-api-access-wjprd" pod "cc64803f-bae9-4ebd-b34e-5667b9eac74a" (UID: "cc64803f-bae9-4ebd-b34e-5667b9eac74a"). InnerVolumeSpecName "kube-api-access-wjprd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:20:51.659817 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:20:51.659730    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc64803f-bae9-4ebd-b34e-5667b9eac74a-dshm" pod "cc64803f-bae9-4ebd-b34e-5667b9eac74a" (UID: "cc64803f-bae9-4ebd-b34e-5667b9eac74a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:20:51.659817 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:20:51.659768    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc64803f-bae9-4ebd-b34e-5667b9eac74a-tls-certs" pod "cc64803f-bae9-4ebd-b34e-5667b9eac74a" (UID: "cc64803f-bae9-4ebd-b34e-5667b9eac74a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:20:51.759392 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:20:51.759352    4521 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjprd\" (UniqueName: \"kubernetes.io/projected/cc64803f-bae9-4ebd-b34e-5667b9eac74a-kube-api-access-wjprd\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:20:51.759392 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:20:51.759386    4521 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc64803f-bae9-4ebd-b34e-5667b9eac74a-dshm\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:20:51.759392 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:20:51.759397    4521 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc64803f-bae9-4ebd-b34e-5667b9eac74a-tls-certs\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:20:52.355081 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:20:52.355044    4521 generic.go:356] "Generic (PLEG): container finished" podID="cc64803f-bae9-4ebd-b34e-5667b9eac74a" containerID="e856922b5c65bff0d88aa35445822fa40136c150d118c9be06b09de812fbc2fa" exitCode=0
Oct 30 21:20:52.355563 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:20:52.355099    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-3f8108d8-kserve-65df7899df-4wpdz" event={"ID":"cc64803f-bae9-4ebd-b34e-5667b9eac74a","Type":"ContainerDied","Data":"e856922b5c65bff0d88aa35445822fa40136c150d118c9be06b09de812fbc2fa"}
Oct 30 21:20:52.355563 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:20:52.355129    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-3f8108d8-kserve-65df7899df-4wpdz" event={"ID":"cc64803f-bae9-4ebd-b34e-5667b9eac74a","Type":"ContainerDied","Data":"db129113cc9197c483951cce43850a237811dadd1ded778f31c22e3d6716c637"}
Oct 30 21:20:52.355563 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:20:52.355145    4521 scope.go:122] "RemoveContainer" containerID="e856922b5c65bff0d88aa35445822fa40136c150d118c9be06b09de812fbc2fa"
Oct 30 21:20:52.355563 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:20:52.355099    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-3f8108d8-kserve-65df7899df-4wpdz"
Oct 30 21:20:52.367927 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:20:52.367904    4521 scope.go:122] "RemoveContainer" containerID="e856922b5c65bff0d88aa35445822fa40136c150d118c9be06b09de812fbc2fa"
Oct 30 21:20:52.368447 ip-10-0-131-125 kubenswrapper[4521]: E1030 21:20:52.368414    4521 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e856922b5c65bff0d88aa35445822fa40136c150d118c9be06b09de812fbc2fa\": container with ID starting with e856922b5c65bff0d88aa35445822fa40136c150d118c9be06b09de812fbc2fa not found: ID does not exist" containerID="e856922b5c65bff0d88aa35445822fa40136c150d118c9be06b09de812fbc2fa"
Oct 30 21:20:52.368551 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:20:52.368460    4521 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e856922b5c65bff0d88aa35445822fa40136c150d118c9be06b09de812fbc2fa"} err="failed to get container status \"e856922b5c65bff0d88aa35445822fa40136c150d118c9be06b09de812fbc2fa\": rpc error: code = NotFound desc = could not find container \"e856922b5c65bff0d88aa35445822fa40136c150d118c9be06b09de812fbc2fa\": container with ID starting with e856922b5c65bff0d88aa35445822fa40136c150d118c9be06b09de812fbc2fa not found: ID does not exist"
Oct 30 21:20:52.384969 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:20:52.384936    4521 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-3f8108d8-kserve-65df7899df-4wpdz"]
Oct 30 21:20:52.389378 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:20:52.389348    4521 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-3f8108d8-kserve-65df7899df-4wpdz"]
Oct 30 21:20:52.408548 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:20:52.408514    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4kxb"]
Oct 30 21:20:52.796432 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:20:52.796397    4521 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc64803f-bae9-4ebd-b34e-5667b9eac74a" path="/var/lib/kubelet/pods/cc64803f-bae9-4ebd-b34e-5667b9eac74a/volumes"
Oct 30 21:20:56.857262 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:20:56.857212    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4kxb"]
Oct 30 21:21:13.118712 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:21:13.118675    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4kxb"]
Oct 30 21:21:16.428386 ip-10-0-131-125 kubenswrapper[4521]: E1030 21:21:16.428347    4521 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lkn2k" containerName="extractor"
Oct 30 21:21:28.825948 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:21:28.825912    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4kxb"]
Oct 30 21:21:40.310726 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:21:40.310687    4521 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-68a46da4-wtx4r"]
Oct 30 21:21:40.311132 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:21:40.311115    4521 cpu_manager.go:414] "containerMap: removing container" podUID="cc64803f-bae9-4ebd-b34e-5667b9eac74a" containerName="main"
Oct 30 21:21:40.311173 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:21:40.311129    4521 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc64803f-bae9-4ebd-b34e-5667b9eac74a" containerName="main"
Oct 30 21:21:40.311208 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:21:40.311198    4521 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc64803f-bae9-4ebd-b34e-5667b9eac74a" containerName="main"
Oct 30 21:21:40.314009 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:21:40.313989    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-68a46da4-wtx4r"
Oct 30 21:21:40.321930 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:21:40.321876    4521 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:21:40.322067 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:21:40.321942    4521 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:21:40.322144 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:21:40.322116    4521 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c45pw\""
Oct 30 21:21:40.322274 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:21:40.322244    4521 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:21:40.324627 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:21:40.324604    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-68a46da4-wtx4r"]
Oct 30 21:21:40.384625 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:21:40.384592    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a540a4b-e0d3-4fa3-85a8-5e29888514d1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-68a46da4-wtx4r\" (UID: \"6a540a4b-e0d3-4fa3-85a8-5e29888514d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-68a46da4-wtx4r"
Oct 30 21:21:40.384789 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:21:40.384626    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knq8s\" (UniqueName: \"kubernetes.io/projected/6a540a4b-e0d3-4fa3-85a8-5e29888514d1-kube-api-access-knq8s\") pod \"maas-api-revoke-keys-e2e-ait-68a46da4-wtx4r\" (UID: \"6a540a4b-e0d3-4fa3-85a8-5e29888514d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-68a46da4-wtx4r"
Oct 30 21:21:40.384789 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:21:40.384650    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a540a4b-e0d3-4fa3-85a8-5e29888514d1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-68a46da4-wtx4r\" (UID: \"6a540a4b-e0d3-4fa3-85a8-5e29888514d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-68a46da4-wtx4r"
Oct 30 21:21:40.485935 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:21:40.485897    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a540a4b-e0d3-4fa3-85a8-5e29888514d1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-68a46da4-wtx4r\" (UID: \"6a540a4b-e0d3-4fa3-85a8-5e29888514d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-68a46da4-wtx4r"
Oct 30 21:21:40.485935 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:21:40.485935    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knq8s\" (UniqueName: \"kubernetes.io/projected/6a540a4b-e0d3-4fa3-85a8-5e29888514d1-kube-api-access-knq8s\") pod \"maas-api-revoke-keys-e2e-ait-68a46da4-wtx4r\" (UID: \"6a540a4b-e0d3-4fa3-85a8-5e29888514d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-68a46da4-wtx4r"
Oct 30 21:21:40.486168 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:21:40.485958    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a540a4b-e0d3-4fa3-85a8-5e29888514d1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-68a46da4-wtx4r\" (UID: \"6a540a4b-e0d3-4fa3-85a8-5e29888514d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-68a46da4-wtx4r"
Oct 30 21:21:40.486741 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:21:40.486713    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a540a4b-e0d3-4fa3-85a8-5e29888514d1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-68a46da4-wtx4r\" (UID: \"6a540a4b-e0d3-4fa3-85a8-5e29888514d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-68a46da4-wtx4r"
Oct 30 21:21:40.488335 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:21:40.488301    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a540a4b-e0d3-4fa3-85a8-5e29888514d1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-68a46da4-wtx4r\" (UID: \"6a540a4b-e0d3-4fa3-85a8-5e29888514d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-68a46da4-wtx4r"
Oct 30 21:21:40.495037 ip-10-0-131-125 kubenswrapper[4521]: E1030 21:21:40.495018    4521 projected.go:196] Error preparing data for projected volume kube-api-access-knq8s for pod odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-68a46da4-wtx4r: failed to fetch token: serviceaccounts "maas-api-cleanup" not found
Oct 30 21:21:40.495141 ip-10-0-131-125 kubenswrapper[4521]: E1030 21:21:40.495100    4521 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6a540a4b-e0d3-4fa3-85a8-5e29888514d1-kube-api-access-knq8s podName:6a540a4b-e0d3-4fa3-85a8-5e29888514d1 nodeName:}" failed. No retries permitted until 2026-10-30 21:21:40.995087636 +0000 UTC m=+2568.340708852 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-knq8s" (UniqueName: "kubernetes.io/projected/6a540a4b-e0d3-4fa3-85a8-5e29888514d1-kube-api-access-knq8s") pod "maas-api-revoke-keys-e2e-ait-68a46da4-wtx4r" (UID: "6a540a4b-e0d3-4fa3-85a8-5e29888514d1") : failed to fetch token: serviceaccounts "maas-api-cleanup" not found
Oct 30 21:21:41.090657 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:21:41.090617    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knq8s\" (UniqueName: \"kubernetes.io/projected/6a540a4b-e0d3-4fa3-85a8-5e29888514d1-kube-api-access-knq8s\") pod \"maas-api-revoke-keys-e2e-ait-68a46da4-wtx4r\" (UID: \"6a540a4b-e0d3-4fa3-85a8-5e29888514d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-68a46da4-wtx4r"
Oct 30 21:21:41.103393 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:21:41.103360    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-knq8s\" (UniqueName: \"kubernetes.io/projected/6a540a4b-e0d3-4fa3-85a8-5e29888514d1-kube-api-access-knq8s\") pod \"maas-api-revoke-keys-e2e-ait-68a46da4-wtx4r\" (UID: \"6a540a4b-e0d3-4fa3-85a8-5e29888514d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-68a46da4-wtx4r"
Oct 30 21:21:41.227489 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:21:41.227448    4521 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-68a46da4-wtx4r" secret="" err="secret \"maas-api-cleanup-dockercfg-c45pw\" not found"
Oct 30 21:21:41.227700 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:21:41.227509    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-68a46da4-wtx4r"
Oct 30 21:21:41.505968 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:21:41.505936    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-68a46da4-wtx4r"]
Oct 30 21:21:41.507385 ip-10-0-131-125 kubenswrapper[4521]: W1030 21:21:41.507357    4521 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a540a4b_e0d3_4fa3_85a8_5e29888514d1.slice/crio-593e61a2ce7381bb21971719e64025b721782d63710b3a1c487e1bd0df59f4d1 WatchSource:0}: Error finding container 593e61a2ce7381bb21971719e64025b721782d63710b3a1c487e1bd0df59f4d1: Status 404 returned error can't find the container with id 593e61a2ce7381bb21971719e64025b721782d63710b3a1c487e1bd0df59f4d1
Oct 30 21:21:41.645076 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:21:41.645042    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-68a46da4-wtx4r" event={"ID":"6a540a4b-e0d3-4fa3-85a8-5e29888514d1","Type":"ContainerStarted","Data":"593e61a2ce7381bb21971719e64025b721782d63710b3a1c487e1bd0df59f4d1"}
Oct 30 21:21:42.653305 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:21:42.653200    4521 generic.go:356] "Generic (PLEG): container finished" podID="6a540a4b-e0d3-4fa3-85a8-5e29888514d1" containerID="e6cd23bf4600bcda5a2d0e658ee95553a0611d5db5af1be51f76ee52e213007e" exitCode=0
Oct 30 21:21:42.653755 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:21:42.653386    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-68a46da4-wtx4r" event={"ID":"6a540a4b-e0d3-4fa3-85a8-5e29888514d1","Type":"ContainerDied","Data":"e6cd23bf4600bcda5a2d0e658ee95553a0611d5db5af1be51f76ee52e213007e"}
Oct 30 21:21:43.788692 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:21:43.788669    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-68a46da4-wtx4r"
Oct 30 21:21:43.914474 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:21:43.914390    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a540a4b-e0d3-4fa3-85a8-5e29888514d1-kube-api-access-knq8s\" (UniqueName: \"kubernetes.io/projected/6a540a4b-e0d3-4fa3-85a8-5e29888514d1-kube-api-access-knq8s\") pod \"6a540a4b-e0d3-4fa3-85a8-5e29888514d1\" (UID: \"6a540a4b-e0d3-4fa3-85a8-5e29888514d1\") "
Oct 30 21:21:43.914474 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:21:43.914471    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a540a4b-e0d3-4fa3-85a8-5e29888514d1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a540a4b-e0d3-4fa3-85a8-5e29888514d1-maas-api-xks-ca\") pod \"6a540a4b-e0d3-4fa3-85a8-5e29888514d1\" (UID: \"6a540a4b-e0d3-4fa3-85a8-5e29888514d1\") "
Oct 30 21:21:43.914721 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:21:43.914496    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a540a4b-e0d3-4fa3-85a8-5e29888514d1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a540a4b-e0d3-4fa3-85a8-5e29888514d1-maas-api-openshift-ca\") pod \"6a540a4b-e0d3-4fa3-85a8-5e29888514d1\" (UID: \"6a540a4b-e0d3-4fa3-85a8-5e29888514d1\") "
Oct 30 21:21:43.914966 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:21:43.914938    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a540a4b-e0d3-4fa3-85a8-5e29888514d1-maas-api-openshift-ca" pod "6a540a4b-e0d3-4fa3-85a8-5e29888514d1" (UID: "6a540a4b-e0d3-4fa3-85a8-5e29888514d1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:21:43.916552 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:21:43.916530    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a540a4b-e0d3-4fa3-85a8-5e29888514d1-maas-api-xks-ca" pod "6a540a4b-e0d3-4fa3-85a8-5e29888514d1" (UID: "6a540a4b-e0d3-4fa3-85a8-5e29888514d1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:21:43.916627 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:21:43.916551    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a540a4b-e0d3-4fa3-85a8-5e29888514d1-kube-api-access-knq8s" pod "6a540a4b-e0d3-4fa3-85a8-5e29888514d1" (UID: "6a540a4b-e0d3-4fa3-85a8-5e29888514d1"). InnerVolumeSpecName "kube-api-access-knq8s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:21:44.016273 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:21:44.016240    4521 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a540a4b-e0d3-4fa3-85a8-5e29888514d1-maas-api-xks-ca\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:21:44.016273 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:21:44.016270    4521 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a540a4b-e0d3-4fa3-85a8-5e29888514d1-maas-api-openshift-ca\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:21:44.016273 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:21:44.016279    4521 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-knq8s\" (UniqueName: \"kubernetes.io/projected/6a540a4b-e0d3-4fa3-85a8-5e29888514d1-kube-api-access-knq8s\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:21:44.666044 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:21:44.665997    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-68a46da4-wtx4r" event={"ID":"6a540a4b-e0d3-4fa3-85a8-5e29888514d1","Type":"ContainerDied","Data":"593e61a2ce7381bb21971719e64025b721782d63710b3a1c487e1bd0df59f4d1"}
Oct 30 21:21:44.666382 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:21:44.666086    4521 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="593e61a2ce7381bb21971719e64025b721782d63710b3a1c487e1bd0df59f4d1"
Oct 30 21:21:44.666382 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:21:44.666022    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-68a46da4-wtx4r"
Oct 30 21:21:46.428798 ip-10-0-131-125 kubenswrapper[4521]: E1030 21:21:46.428761    4521 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lkn2k" containerName="extractor"
Oct 30 21:21:50.946602 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:21:50.946561    4521 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-68a46da4-wtx4r"]
Oct 30 21:21:50.950132 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:21:50.950105    4521 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-68a46da4-wtx4r"]
Oct 30 21:21:52.192941 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:21:52.192902    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4kxb"]
Oct 30 21:21:52.790868 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:21:52.790837    4521 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a540a4b-e0d3-4fa3-85a8-5e29888514d1" path="/var/lib/kubelet/pods/6a540a4b-e0d3-4fa3-85a8-5e29888514d1/volumes"
Oct 30 21:21:54.816552 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:21:54.816512    4521 scope.go:122] "RemoveContainer" containerID="1fd89f488e23fe6f3aba55c1ec6110f11acffd994306c75398b8b4486cc71e83"
Oct 30 21:21:54.827242 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:21:54.827218    4521 scope.go:122] "RemoveContainer" containerID="f80614565e3c6f468f3662eb052d2d1d3f61884265918f1431df49e3be632bc0"
Oct 30 21:22:06.015099 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:22:06.015053    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4kxb"]
Oct 30 21:22:16.429193 ip-10-0-131-125 kubenswrapper[4521]: E1030 21:22:16.429155    4521 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lkn2k" containerName="extractor"
Oct 30 21:22:46.429711 ip-10-0-131-125 kubenswrapper[4521]: E1030 21:22:46.429670    4521 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lkn2k" containerName="extractor"
Oct 30 21:23:16.432142 ip-10-0-131-125 kubenswrapper[4521]: E1030 21:23:16.432054    4521 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lkn2k" containerName="extractor"
Oct 30 21:23:39.407556 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:23:39.407519    4521 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-d8974d-64m2d"]
Oct 30 21:23:39.408034 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:23:39.407902    4521 cpu_manager.go:414] "containerMap: removing container" podUID="6a540a4b-e0d3-4fa3-85a8-5e29888514d1" containerName="revoke-keys"
Oct 30 21:23:39.408034 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:23:39.407913    4521 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a540a4b-e0d3-4fa3-85a8-5e29888514d1" containerName="revoke-keys"
Oct 30 21:23:39.408034 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:23:39.407978    4521 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a540a4b-e0d3-4fa3-85a8-5e29888514d1" containerName="revoke-keys"
Oct 30 21:23:39.410915 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:23:39.410899    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-d8974d-64m2d"
Oct 30 21:23:39.418418 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:23:39.418398    4521 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:23:39.419256 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:23:39.418650    4521 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vtzj4\""
Oct 30 21:23:39.419256 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:23:39.418729    4521 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:23:39.419256 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:23:39.418755    4521 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:23:39.420471 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:23:39.420450    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-d8974d-64m2d"]
Oct 30 21:23:39.564587 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:23:39.564545    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15d19cca-1f54-4dba-9a44-3982b3ed06c0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-d8974d-64m2d\" (UID: \"15d19cca-1f54-4dba-9a44-3982b3ed06c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-d8974d-64m2d"
Oct 30 21:23:39.564772 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:23:39.564593    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15d19cca-1f54-4dba-9a44-3982b3ed06c0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-d8974d-64m2d\" (UID: \"15d19cca-1f54-4dba-9a44-3982b3ed06c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-d8974d-64m2d"
Oct 30 21:23:39.564772 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:23:39.564720    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndm5c\" (UniqueName: \"kubernetes.io/projected/15d19cca-1f54-4dba-9a44-3982b3ed06c0-kube-api-access-ndm5c\") pod \"maas-api-revoke-keys-e2e-authcfg-http-d8974d-64m2d\" (UID: \"15d19cca-1f54-4dba-9a44-3982b3ed06c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-d8974d-64m2d"
Oct 30 21:23:39.665705 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:23:39.665621    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15d19cca-1f54-4dba-9a44-3982b3ed06c0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-d8974d-64m2d\" (UID: \"15d19cca-1f54-4dba-9a44-3982b3ed06c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-d8974d-64m2d"
Oct 30 21:23:39.665705 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:23:39.665660    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15d19cca-1f54-4dba-9a44-3982b3ed06c0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-d8974d-64m2d\" (UID: \"15d19cca-1f54-4dba-9a44-3982b3ed06c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-d8974d-64m2d"
Oct 30 21:23:39.665915 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:23:39.665714    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndm5c\" (UniqueName: \"kubernetes.io/projected/15d19cca-1f54-4dba-9a44-3982b3ed06c0-kube-api-access-ndm5c\") pod \"maas-api-revoke-keys-e2e-authcfg-http-d8974d-64m2d\" (UID: \"15d19cca-1f54-4dba-9a44-3982b3ed06c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-d8974d-64m2d"
Oct 30 21:23:39.666469 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:23:39.666446    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15d19cca-1f54-4dba-9a44-3982b3ed06c0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-d8974d-64m2d\" (UID: \"15d19cca-1f54-4dba-9a44-3982b3ed06c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-d8974d-64m2d"
Oct 30 21:23:39.668002 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:23:39.667976    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15d19cca-1f54-4dba-9a44-3982b3ed06c0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-d8974d-64m2d\" (UID: \"15d19cca-1f54-4dba-9a44-3982b3ed06c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-d8974d-64m2d"
Oct 30 21:23:39.679569 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:23:39.679539    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndm5c\" (UniqueName: \"kubernetes.io/projected/15d19cca-1f54-4dba-9a44-3982b3ed06c0-kube-api-access-ndm5c\") pod \"maas-api-revoke-keys-e2e-authcfg-http-d8974d-64m2d\" (UID: \"15d19cca-1f54-4dba-9a44-3982b3ed06c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-d8974d-64m2d"
Oct 30 21:23:39.726484 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:23:39.726444    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-d8974d-64m2d"
Oct 30 21:23:40.001272 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:23:40.001243    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-d8974d-64m2d"]
Oct 30 21:23:40.002817 ip-10-0-131-125 kubenswrapper[4521]: W1030 21:23:40.002788    4521 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15d19cca_1f54_4dba_9a44_3982b3ed06c0.slice/crio-59e3d1528d378b34c2cbc59c61e5645fdc5eddfb6322dca484f1f19bd72fa619 WatchSource:0}: Error finding container 59e3d1528d378b34c2cbc59c61e5645fdc5eddfb6322dca484f1f19bd72fa619: Status 404 returned error can't find the container with id 59e3d1528d378b34c2cbc59c61e5645fdc5eddfb6322dca484f1f19bd72fa619
Oct 30 21:23:40.277005 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:23:40.276913    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-d8974d-64m2d" event={"ID":"15d19cca-1f54-4dba-9a44-3982b3ed06c0","Type":"ContainerStarted","Data":"6fef15f7f38dd40cfcaf2899d54c7fd206d7c3b19e6a28346daa209550b7a55f"}
Oct 30 21:23:40.277005 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:23:40.276966    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-d8974d-64m2d" event={"ID":"15d19cca-1f54-4dba-9a44-3982b3ed06c0","Type":"ContainerStarted","Data":"59e3d1528d378b34c2cbc59c61e5645fdc5eddfb6322dca484f1f19bd72fa619"}
Oct 30 21:23:40.296943 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:23:40.296896    4521 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-d8974d-64m2d" podStartSLOduration=1.296883021 podStartE2EDuration="1.296883021s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:23:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:23:40.295243294 +0000 UTC m=+2687.640864532" watchObservedRunningTime="2026-10-30 21:23:40.296883021 +0000 UTC m=+2687.642504260"
Oct 30 21:23:41.283896 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:23:41.283855    4521 generic.go:356] "Generic (PLEG): container finished" podID="15d19cca-1f54-4dba-9a44-3982b3ed06c0" containerID="6fef15f7f38dd40cfcaf2899d54c7fd206d7c3b19e6a28346daa209550b7a55f" exitCode=0
Oct 30 21:23:41.284304 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:23:41.283928    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-d8974d-64m2d" event={"ID":"15d19cca-1f54-4dba-9a44-3982b3ed06c0","Type":"ContainerDied","Data":"6fef15f7f38dd40cfcaf2899d54c7fd206d7c3b19e6a28346daa209550b7a55f"}
Oct 30 21:23:42.420117 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:23:42.420089    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-d8974d-64m2d"
Oct 30 21:23:42.592886 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:23:42.592797    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15d19cca-1f54-4dba-9a44-3982b3ed06c0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15d19cca-1f54-4dba-9a44-3982b3ed06c0-maas-api-openshift-ca\") pod \"15d19cca-1f54-4dba-9a44-3982b3ed06c0\" (UID: \"15d19cca-1f54-4dba-9a44-3982b3ed06c0\") "
Oct 30 21:23:42.592886 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:23:42.592886    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15d19cca-1f54-4dba-9a44-3982b3ed06c0-kube-api-access-ndm5c\" (UniqueName: \"kubernetes.io/projected/15d19cca-1f54-4dba-9a44-3982b3ed06c0-kube-api-access-ndm5c\") pod \"15d19cca-1f54-4dba-9a44-3982b3ed06c0\" (UID: \"15d19cca-1f54-4dba-9a44-3982b3ed06c0\") "
Oct 30 21:23:42.593097 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:23:42.592917    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/15d19cca-1f54-4dba-9a44-3982b3ed06c0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15d19cca-1f54-4dba-9a44-3982b3ed06c0-maas-api-xks-ca\") pod \"15d19cca-1f54-4dba-9a44-3982b3ed06c0\" (UID: \"15d19cca-1f54-4dba-9a44-3982b3ed06c0\") "
Oct 30 21:23:42.593184 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:23:42.593160    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15d19cca-1f54-4dba-9a44-3982b3ed06c0-maas-api-openshift-ca" pod "15d19cca-1f54-4dba-9a44-3982b3ed06c0" (UID: "15d19cca-1f54-4dba-9a44-3982b3ed06c0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:23:42.595077 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:23:42.595048    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15d19cca-1f54-4dba-9a44-3982b3ed06c0-maas-api-xks-ca" pod "15d19cca-1f54-4dba-9a44-3982b3ed06c0" (UID: "15d19cca-1f54-4dba-9a44-3982b3ed06c0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:23:42.595188 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:23:42.595072    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15d19cca-1f54-4dba-9a44-3982b3ed06c0-kube-api-access-ndm5c" pod "15d19cca-1f54-4dba-9a44-3982b3ed06c0" (UID: "15d19cca-1f54-4dba-9a44-3982b3ed06c0"). InnerVolumeSpecName "kube-api-access-ndm5c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:23:42.693747 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:23:42.693713    4521 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15d19cca-1f54-4dba-9a44-3982b3ed06c0-maas-api-openshift-ca\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:42.693747 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:23:42.693742    4521 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ndm5c\" (UniqueName: \"kubernetes.io/projected/15d19cca-1f54-4dba-9a44-3982b3ed06c0-kube-api-access-ndm5c\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:42.693747 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:23:42.693753    4521 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15d19cca-1f54-4dba-9a44-3982b3ed06c0-maas-api-xks-ca\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:43.296288 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:23:43.296255    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-d8974d-64m2d"
Oct 30 21:23:43.296499 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:23:43.296287    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-d8974d-64m2d" event={"ID":"15d19cca-1f54-4dba-9a44-3982b3ed06c0","Type":"ContainerDied","Data":"59e3d1528d378b34c2cbc59c61e5645fdc5eddfb6322dca484f1f19bd72fa619"}
Oct 30 21:23:43.296499 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:23:43.296363    4521 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="59e3d1528d378b34c2cbc59c61e5645fdc5eddfb6322dca484f1f19bd72fa619"
Oct 30 21:23:46.426915 ip-10-0-131-125 kubenswrapper[4521]: E1030 21:23:46.426880    4521 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lkn2k" containerName="extractor"
Oct 30 21:23:55.797183 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:23:55.797146    4521 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-d8974d-64m2d"]
Oct 30 21:23:55.802169 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:23:55.802142    4521 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-d8974d-64m2d"]
Oct 30 21:23:56.791982 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:23:56.791947    4521 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15d19cca-1f54-4dba-9a44-3982b3ed06c0" path="/var/lib/kubelet/pods/15d19cca-1f54-4dba-9a44-3982b3ed06c0/volumes"
Oct 30 21:23:57.113580 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:23:57.113494    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4kxb"]
Oct 30 21:24:04.899583 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:24:04.899544    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4kxb"]
Oct 30 21:24:10.201339 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:24:10.201296    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4kxb"]
Oct 30 21:24:14.775560 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:24:14.775524    4521 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-nsg9r"]
Oct 30 21:24:14.776041 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:24:14.775871    4521 cpu_manager.go:414] "containerMap: removing container" podUID="15d19cca-1f54-4dba-9a44-3982b3ed06c0" containerName="revoke-keys"
Oct 30 21:24:14.776041 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:24:14.775881    4521 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15d19cca-1f54-4dba-9a44-3982b3ed06c0" containerName="revoke-keys"
Oct 30 21:24:14.776041 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:24:14.775965    4521 memory_manager.go:365] "RemoveStaleState removing state" podUID="15d19cca-1f54-4dba-9a44-3982b3ed06c0" containerName="revoke-keys"
Oct 30 21:24:14.780042 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:24:14.780026    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-nsg9r"
Oct 30 21:24:14.790117 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:24:14.787921    4521 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:24:14.790117 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:24:14.787932    4521 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:24:14.790117 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:24:14.787974    4521 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:24:14.790117 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:24:14.788011    4521 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vtzj4\""
Oct 30 21:24:14.793157 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:24:14.793130    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-nsg9r"]
Oct 30 21:24:14.816576 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:24:14.816533    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a95d0b7-c673-45ee-b23d-e958e79a80a5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-nsg9r\" (UID: \"0a95d0b7-c673-45ee-b23d-e958e79a80a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-nsg9r"
Oct 30 21:24:14.816754 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:24:14.816621    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-866ns\" (UniqueName: \"kubernetes.io/projected/0a95d0b7-c673-45ee-b23d-e958e79a80a5-kube-api-access-866ns\") pod \"maas-api-revoke-sub-f03629db-nsg9r\" (UID: \"0a95d0b7-c673-45ee-b23d-e958e79a80a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-nsg9r"
Oct 30 21:24:14.816754 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:24:14.816725    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a95d0b7-c673-45ee-b23d-e958e79a80a5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-nsg9r\" (UID: \"0a95d0b7-c673-45ee-b23d-e958e79a80a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-nsg9r"
Oct 30 21:24:14.917798 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:24:14.917763    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-866ns\" (UniqueName: \"kubernetes.io/projected/0a95d0b7-c673-45ee-b23d-e958e79a80a5-kube-api-access-866ns\") pod \"maas-api-revoke-sub-f03629db-nsg9r\" (UID: \"0a95d0b7-c673-45ee-b23d-e958e79a80a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-nsg9r"
Oct 30 21:24:14.917798 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:24:14.917800    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a95d0b7-c673-45ee-b23d-e958e79a80a5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-nsg9r\" (UID: \"0a95d0b7-c673-45ee-b23d-e958e79a80a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-nsg9r"
Oct 30 21:24:14.918019 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:24:14.917873    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a95d0b7-c673-45ee-b23d-e958e79a80a5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-nsg9r\" (UID: \"0a95d0b7-c673-45ee-b23d-e958e79a80a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-nsg9r"
Oct 30 21:24:14.918550 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:24:14.918528    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a95d0b7-c673-45ee-b23d-e958e79a80a5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-nsg9r\" (UID: \"0a95d0b7-c673-45ee-b23d-e958e79a80a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-nsg9r"
Oct 30 21:24:14.920358 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:24:14.920318    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a95d0b7-c673-45ee-b23d-e958e79a80a5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-nsg9r\" (UID: \"0a95d0b7-c673-45ee-b23d-e958e79a80a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-nsg9r"
Oct 30 21:24:14.930149 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:24:14.930126    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-866ns\" (UniqueName: \"kubernetes.io/projected/0a95d0b7-c673-45ee-b23d-e958e79a80a5-kube-api-access-866ns\") pod \"maas-api-revoke-sub-f03629db-nsg9r\" (UID: \"0a95d0b7-c673-45ee-b23d-e958e79a80a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-nsg9r"
Oct 30 21:24:15.097225 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:24:15.097136    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-nsg9r"
Oct 30 21:24:15.375547 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:24:15.375517    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-nsg9r"]
Oct 30 21:24:15.484825 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:24:15.484790    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-nsg9r" event={"ID":"0a95d0b7-c673-45ee-b23d-e958e79a80a5","Type":"ContainerStarted","Data":"a3eab6ed1652e06e650a6ae56d460c78c4dae289720cc6f6e1a6674d0ecf2969"}
Oct 30 21:24:16.428574 ip-10-0-131-125 kubenswrapper[4521]: E1030 21:24:16.428491    4521 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lkn2k" containerName="extractor"
Oct 30 21:24:16.491491 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:24:16.491460    4521 generic.go:356] "Generic (PLEG): container finished" podID="0a95d0b7-c673-45ee-b23d-e958e79a80a5" containerID="76be077aec642adab8d5903298619e9d77176634bca312a8f1eb6abbb96d29dc" exitCode=0
Oct 30 21:24:16.491680 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:24:16.491519    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-nsg9r" event={"ID":"0a95d0b7-c673-45ee-b23d-e958e79a80a5","Type":"ContainerDied","Data":"76be077aec642adab8d5903298619e9d77176634bca312a8f1eb6abbb96d29dc"}
Oct 30 21:24:17.624868 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:24:17.624839    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-nsg9r"
Oct 30 21:24:17.745313 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:24:17.745275    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a95d0b7-c673-45ee-b23d-e958e79a80a5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a95d0b7-c673-45ee-b23d-e958e79a80a5-maas-api-openshift-ca\") pod \"0a95d0b7-c673-45ee-b23d-e958e79a80a5\" (UID: \"0a95d0b7-c673-45ee-b23d-e958e79a80a5\") "
Oct 30 21:24:17.745313 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:24:17.745314    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a95d0b7-c673-45ee-b23d-e958e79a80a5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a95d0b7-c673-45ee-b23d-e958e79a80a5-maas-api-xks-ca\") pod \"0a95d0b7-c673-45ee-b23d-e958e79a80a5\" (UID: \"0a95d0b7-c673-45ee-b23d-e958e79a80a5\") "
Oct 30 21:24:17.745602 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:24:17.745452    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a95d0b7-c673-45ee-b23d-e958e79a80a5-kube-api-access-866ns\" (UniqueName: \"kubernetes.io/projected/0a95d0b7-c673-45ee-b23d-e958e79a80a5-kube-api-access-866ns\") pod \"0a95d0b7-c673-45ee-b23d-e958e79a80a5\" (UID: \"0a95d0b7-c673-45ee-b23d-e958e79a80a5\") "
Oct 30 21:24:17.745747 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:24:17.745719    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a95d0b7-c673-45ee-b23d-e958e79a80a5-maas-api-openshift-ca" pod "0a95d0b7-c673-45ee-b23d-e958e79a80a5" (UID: "0a95d0b7-c673-45ee-b23d-e958e79a80a5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:24:17.747630 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:24:17.747566    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a95d0b7-c673-45ee-b23d-e958e79a80a5-maas-api-xks-ca" pod "0a95d0b7-c673-45ee-b23d-e958e79a80a5" (UID: "0a95d0b7-c673-45ee-b23d-e958e79a80a5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:24:17.747720 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:24:17.747625    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a95d0b7-c673-45ee-b23d-e958e79a80a5-kube-api-access-866ns" pod "0a95d0b7-c673-45ee-b23d-e958e79a80a5" (UID: "0a95d0b7-c673-45ee-b23d-e958e79a80a5"). InnerVolumeSpecName "kube-api-access-866ns". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:24:17.846637 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:24:17.846598    4521 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-866ns\" (UniqueName: \"kubernetes.io/projected/0a95d0b7-c673-45ee-b23d-e958e79a80a5-kube-api-access-866ns\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:24:17.846637 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:24:17.846629    4521 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a95d0b7-c673-45ee-b23d-e958e79a80a5-maas-api-openshift-ca\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:24:17.846637 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:24:17.846640    4521 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a95d0b7-c673-45ee-b23d-e958e79a80a5-maas-api-xks-ca\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:24:18.502983 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:24:18.502946    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-nsg9r" event={"ID":"0a95d0b7-c673-45ee-b23d-e958e79a80a5","Type":"ContainerDied","Data":"a3eab6ed1652e06e650a6ae56d460c78c4dae289720cc6f6e1a6674d0ecf2969"}
Oct 30 21:24:18.503156 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:24:18.503006    4521 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a3eab6ed1652e06e650a6ae56d460c78c4dae289720cc6f6e1a6674d0ecf2969"
Oct 30 21:24:18.503156 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:24:18.502966    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-nsg9r"
Oct 30 21:24:21.022459 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:24:21.022421    4521 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3dc5db79-r99pk"]
Oct 30 21:24:21.025477 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:24:21.025447    4521 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3dc5db79-r99pk"]
Oct 30 21:24:22.796604 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:24:22.796572    4521 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18b7bc42-cacf-43cd-92f0-d600df31689e" path="/var/lib/kubelet/pods/18b7bc42-cacf-43cd-92f0-d600df31689e/volumes"
Oct 30 21:24:25.584028 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:24:25.583984    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4kxb"]
Oct 30 21:24:29.973350 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:24:29.965989    4521 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-bckjb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.26:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:24:29.973350 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:24:29.966075    4521 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-bckjb" podUID="62149acf-8605-4f8d-b1ac-6e8d6b4865fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.26:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:24:30.741283 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:24:30.741243    4521 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-tvkpg"]
Oct 30 21:24:30.741654 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:24:30.741640    4521 cpu_manager.go:414] "containerMap: removing container" podUID="0a95d0b7-c673-45ee-b23d-e958e79a80a5" containerName="revoke-keys"
Oct 30 21:24:30.741716 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:24:30.741654    4521 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a95d0b7-c673-45ee-b23d-e958e79a80a5" containerName="revoke-keys"
Oct 30 21:24:30.741762 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:24:30.741716    4521 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a95d0b7-c673-45ee-b23d-e958e79a80a5" containerName="revoke-keys"
Oct 30 21:24:30.744861 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:24:30.744844    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-tvkpg"
Oct 30 21:24:30.748786 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:24:30.748756    4521 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:24:30.749875 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:24:30.749857    4521 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:24:30.749976 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:24:30.749956    4521 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vtzj4\""
Oct 30 21:24:30.750025 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:24:30.749989    4521 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:24:30.756040 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:24:30.756015    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-tvkpg"]
Oct 30 21:24:30.890702 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:24:30.890663    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0549520a-091d-4e1c-bc20-ea430909f25e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-tvkpg\" (UID: \"0549520a-091d-4e1c-bc20-ea430909f25e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-tvkpg"
Oct 30 21:24:30.890702 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:24:30.890714    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6dd7\" (UniqueName: \"kubernetes.io/projected/0549520a-091d-4e1c-bc20-ea430909f25e-kube-api-access-r6dd7\") pod \"maas-api-revoke-sub-0165d7d2-tvkpg\" (UID: \"0549520a-091d-4e1c-bc20-ea430909f25e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-tvkpg"
Oct 30 21:24:30.890925 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:24:30.890891    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0549520a-091d-4e1c-bc20-ea430909f25e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-tvkpg\" (UID: \"0549520a-091d-4e1c-bc20-ea430909f25e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-tvkpg"
Oct 30 21:24:30.992555 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:24:30.992459    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0549520a-091d-4e1c-bc20-ea430909f25e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-tvkpg\" (UID: \"0549520a-091d-4e1c-bc20-ea430909f25e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-tvkpg"
Oct 30 21:24:30.992555 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:24:30.992516    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6dd7\" (UniqueName: \"kubernetes.io/projected/0549520a-091d-4e1c-bc20-ea430909f25e-kube-api-access-r6dd7\") pod \"maas-api-revoke-sub-0165d7d2-tvkpg\" (UID: \"0549520a-091d-4e1c-bc20-ea430909f25e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-tvkpg"
Oct 30 21:24:30.993110 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:24:30.992591    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0549520a-091d-4e1c-bc20-ea430909f25e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-tvkpg\" (UID: \"0549520a-091d-4e1c-bc20-ea430909f25e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-tvkpg"
Oct 30 21:24:30.993390 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:24:30.993367    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0549520a-091d-4e1c-bc20-ea430909f25e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-tvkpg\" (UID: \"0549520a-091d-4e1c-bc20-ea430909f25e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-tvkpg"
Oct 30 21:24:30.994826 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:24:30.994804    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0549520a-091d-4e1c-bc20-ea430909f25e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-tvkpg\" (UID: \"0549520a-091d-4e1c-bc20-ea430909f25e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-tvkpg"
Oct 30 21:24:31.004984 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:24:31.004961    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6dd7\" (UniqueName: \"kubernetes.io/projected/0549520a-091d-4e1c-bc20-ea430909f25e-kube-api-access-r6dd7\") pod \"maas-api-revoke-sub-0165d7d2-tvkpg\" (UID: \"0549520a-091d-4e1c-bc20-ea430909f25e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-tvkpg"
Oct 30 21:24:31.059057 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:24:31.059023    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-tvkpg"
Oct 30 21:24:31.335366 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:24:31.335339    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-tvkpg"]
Oct 30 21:24:31.337086 ip-10-0-131-125 kubenswrapper[4521]: W1030 21:24:31.337059    4521 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0549520a_091d_4e1c_bc20_ea430909f25e.slice/crio-3104be5df62895119f0f5682622df12761adf4e00711d301db6c67019cb64dbd WatchSource:0}: Error finding container 3104be5df62895119f0f5682622df12761adf4e00711d301db6c67019cb64dbd: Status 404 returned error can't find the container with id 3104be5df62895119f0f5682622df12761adf4e00711d301db6c67019cb64dbd
Oct 30 21:24:31.601591 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:24:31.601516    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-tvkpg" event={"ID":"0549520a-091d-4e1c-bc20-ea430909f25e","Type":"ContainerStarted","Data":"3104be5df62895119f0f5682622df12761adf4e00711d301db6c67019cb64dbd"}
Oct 30 21:24:32.608042 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:24:32.608005    4521 generic.go:356] "Generic (PLEG): container finished" podID="0549520a-091d-4e1c-bc20-ea430909f25e" containerID="a56026742cccdea98f60dea283dfe5337906605f63dd05fb3a4ead446daabe80" exitCode=0
Oct 30 21:24:32.608431 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:24:32.608087    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-tvkpg" event={"ID":"0549520a-091d-4e1c-bc20-ea430909f25e","Type":"ContainerDied","Data":"a56026742cccdea98f60dea283dfe5337906605f63dd05fb3a4ead446daabe80"}
Oct 30 21:24:33.739796 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:24:33.739771    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-tvkpg"
Oct 30 21:24:33.817845 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:24:33.817812    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0549520a-091d-4e1c-bc20-ea430909f25e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0549520a-091d-4e1c-bc20-ea430909f25e-maas-api-xks-ca\") pod \"0549520a-091d-4e1c-bc20-ea430909f25e\" (UID: \"0549520a-091d-4e1c-bc20-ea430909f25e\") "
Oct 30 21:24:33.818024 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:24:33.817856    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0549520a-091d-4e1c-bc20-ea430909f25e-kube-api-access-r6dd7\" (UniqueName: \"kubernetes.io/projected/0549520a-091d-4e1c-bc20-ea430909f25e-kube-api-access-r6dd7\") pod \"0549520a-091d-4e1c-bc20-ea430909f25e\" (UID: \"0549520a-091d-4e1c-bc20-ea430909f25e\") "
Oct 30 21:24:33.818024 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:24:33.817964    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0549520a-091d-4e1c-bc20-ea430909f25e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0549520a-091d-4e1c-bc20-ea430909f25e-maas-api-openshift-ca\") pod \"0549520a-091d-4e1c-bc20-ea430909f25e\" (UID: \"0549520a-091d-4e1c-bc20-ea430909f25e\") "
Oct 30 21:24:33.818296 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:24:33.818271    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0549520a-091d-4e1c-bc20-ea430909f25e-maas-api-openshift-ca" pod "0549520a-091d-4e1c-bc20-ea430909f25e" (UID: "0549520a-091d-4e1c-bc20-ea430909f25e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:24:33.819930 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:24:33.819903    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0549520a-091d-4e1c-bc20-ea430909f25e-maas-api-xks-ca" pod "0549520a-091d-4e1c-bc20-ea430909f25e" (UID: "0549520a-091d-4e1c-bc20-ea430909f25e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:24:33.820055 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:24:33.819970    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0549520a-091d-4e1c-bc20-ea430909f25e-kube-api-access-r6dd7" pod "0549520a-091d-4e1c-bc20-ea430909f25e" (UID: "0549520a-091d-4e1c-bc20-ea430909f25e"). InnerVolumeSpecName "kube-api-access-r6dd7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:24:33.919262 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:24:33.919181    4521 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0549520a-091d-4e1c-bc20-ea430909f25e-maas-api-openshift-ca\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:24:33.919262 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:24:33.919212    4521 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0549520a-091d-4e1c-bc20-ea430909f25e-maas-api-xks-ca\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:24:33.919262 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:24:33.919221    4521 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6dd7\" (UniqueName: \"kubernetes.io/projected/0549520a-091d-4e1c-bc20-ea430909f25e-kube-api-access-r6dd7\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:24:34.620163 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:24:34.620131    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-tvkpg"
Oct 30 21:24:34.620372 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:24:34.620164    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-tvkpg" event={"ID":"0549520a-091d-4e1c-bc20-ea430909f25e","Type":"ContainerDied","Data":"3104be5df62895119f0f5682622df12761adf4e00711d301db6c67019cb64dbd"}
Oct 30 21:24:34.620372 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:24:34.620212    4521 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3104be5df62895119f0f5682622df12761adf4e00711d301db6c67019cb64dbd"
Oct 30 21:24:41.510363 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:24:41.510310    4521 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-lm4gg"]
Oct 30 21:24:41.510763 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:24:41.510693    4521 cpu_manager.go:414] "containerMap: removing container" podUID="0549520a-091d-4e1c-bc20-ea430909f25e" containerName="revoke-keys"
Oct 30 21:24:41.510763 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:24:41.510705    4521 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0549520a-091d-4e1c-bc20-ea430909f25e" containerName="revoke-keys"
Oct 30 21:24:41.510838 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:24:41.510764    4521 memory_manager.go:365] "RemoveStaleState removing state" podUID="0549520a-091d-4e1c-bc20-ea430909f25e" containerName="revoke-keys"
Oct 30 21:24:41.513988 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:24:41.513965    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-lm4gg"
Oct 30 21:24:41.520811 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:24:41.520773    4521 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:24:41.520987 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:24:41.520916    4521 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vtzj4\""
Oct 30 21:24:41.526270 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:24:41.526228    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-lm4gg"]
Oct 30 21:24:41.528370 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:24:41.528318    4521 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:24:41.528507 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:24:41.528486    4521 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:24:41.689352 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:24:41.689295    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggsgc\" (UniqueName: \"kubernetes.io/projected/e85be7c5-53ae-4992-a98b-b1842269bb42-kube-api-access-ggsgc\") pod \"maas-api-revoke-sub-6b19338f-lm4gg\" (UID: \"e85be7c5-53ae-4992-a98b-b1842269bb42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-lm4gg"
Oct 30 21:24:41.689532 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:24:41.689362    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e85be7c5-53ae-4992-a98b-b1842269bb42-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-lm4gg\" (UID: \"e85be7c5-53ae-4992-a98b-b1842269bb42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-lm4gg"
Oct 30 21:24:41.689691 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:24:41.689540    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e85be7c5-53ae-4992-a98b-b1842269bb42-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-lm4gg\" (UID: \"e85be7c5-53ae-4992-a98b-b1842269bb42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-lm4gg"
Oct 30 21:24:41.790441 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:24:41.790342    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e85be7c5-53ae-4992-a98b-b1842269bb42-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-lm4gg\" (UID: \"e85be7c5-53ae-4992-a98b-b1842269bb42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-lm4gg"
Oct 30 21:24:41.790441 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:24:41.790430    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggsgc\" (UniqueName: \"kubernetes.io/projected/e85be7c5-53ae-4992-a98b-b1842269bb42-kube-api-access-ggsgc\") pod \"maas-api-revoke-sub-6b19338f-lm4gg\" (UID: \"e85be7c5-53ae-4992-a98b-b1842269bb42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-lm4gg"
Oct 30 21:24:41.790697 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:24:41.790454    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e85be7c5-53ae-4992-a98b-b1842269bb42-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-lm4gg\" (UID: \"e85be7c5-53ae-4992-a98b-b1842269bb42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-lm4gg"
Oct 30 21:24:41.791038 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:24:41.791004    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e85be7c5-53ae-4992-a98b-b1842269bb42-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-lm4gg\" (UID: \"e85be7c5-53ae-4992-a98b-b1842269bb42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-lm4gg"
Oct 30 21:24:41.792973 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:24:41.792952    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e85be7c5-53ae-4992-a98b-b1842269bb42-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-lm4gg\" (UID: \"e85be7c5-53ae-4992-a98b-b1842269bb42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-lm4gg"
Oct 30 21:24:41.803023 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:24:41.802988    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggsgc\" (UniqueName: \"kubernetes.io/projected/e85be7c5-53ae-4992-a98b-b1842269bb42-kube-api-access-ggsgc\") pod \"maas-api-revoke-sub-6b19338f-lm4gg\" (UID: \"e85be7c5-53ae-4992-a98b-b1842269bb42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-lm4gg"
Oct 30 21:24:41.831982 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:24:41.831953    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-lm4gg"
Oct 30 21:24:42.112815 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:24:42.112779    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-lm4gg"]
Oct 30 21:24:42.113569 ip-10-0-131-125 kubenswrapper[4521]: W1030 21:24:42.113537    4521 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode85be7c5_53ae_4992_a98b_b1842269bb42.slice/crio-cd8b0059bc241ec63f827f4819b2d74ebc4e5a1385d1c41fa6a9b29367c2361c WatchSource:0}: Error finding container cd8b0059bc241ec63f827f4819b2d74ebc4e5a1385d1c41fa6a9b29367c2361c: Status 404 returned error can't find the container with id cd8b0059bc241ec63f827f4819b2d74ebc4e5a1385d1c41fa6a9b29367c2361c
Oct 30 21:24:42.663973 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:24:42.663934    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-lm4gg" event={"ID":"e85be7c5-53ae-4992-a98b-b1842269bb42","Type":"ContainerStarted","Data":"1b1a6307512e5c6a5d37622010e70c88ae47cc10451286234c66e403ec11ba35"}
Oct 30 21:24:42.664459 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:24:42.663994    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-lm4gg" event={"ID":"e85be7c5-53ae-4992-a98b-b1842269bb42","Type":"ContainerStarted","Data":"cd8b0059bc241ec63f827f4819b2d74ebc4e5a1385d1c41fa6a9b29367c2361c"}
Oct 30 21:24:42.682762 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:24:42.682664    4521 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-lm4gg" podStartSLOduration=1.682652133 podStartE2EDuration="1.682652133s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:24:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:24:42.681243272 +0000 UTC m=+2750.026864510" watchObservedRunningTime="2026-10-30 21:24:42.682652133 +0000 UTC m=+2750.028273371"
Oct 30 21:24:43.670444 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:24:43.670408    4521 generic.go:356] "Generic (PLEG): container finished" podID="e85be7c5-53ae-4992-a98b-b1842269bb42" containerID="1b1a6307512e5c6a5d37622010e70c88ae47cc10451286234c66e403ec11ba35" exitCode=0
Oct 30 21:24:43.670858 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:24:43.670458    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-lm4gg" event={"ID":"e85be7c5-53ae-4992-a98b-b1842269bb42","Type":"ContainerDied","Data":"1b1a6307512e5c6a5d37622010e70c88ae47cc10451286234c66e403ec11ba35"}
Oct 30 21:24:44.804037 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:24:44.804013    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-lm4gg"
Oct 30 21:24:44.922811 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:24:44.922773    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e85be7c5-53ae-4992-a98b-b1842269bb42-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e85be7c5-53ae-4992-a98b-b1842269bb42-maas-api-xks-ca\") pod \"e85be7c5-53ae-4992-a98b-b1842269bb42\" (UID: \"e85be7c5-53ae-4992-a98b-b1842269bb42\") "
Oct 30 21:24:44.922988 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:24:44.922948    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e85be7c5-53ae-4992-a98b-b1842269bb42-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e85be7c5-53ae-4992-a98b-b1842269bb42-maas-api-openshift-ca\") pod \"e85be7c5-53ae-4992-a98b-b1842269bb42\" (UID: \"e85be7c5-53ae-4992-a98b-b1842269bb42\") "
Oct 30 21:24:44.923056 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:24:44.923017    4521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e85be7c5-53ae-4992-a98b-b1842269bb42-kube-api-access-ggsgc\" (UniqueName: \"kubernetes.io/projected/e85be7c5-53ae-4992-a98b-b1842269bb42-kube-api-access-ggsgc\") pod \"e85be7c5-53ae-4992-a98b-b1842269bb42\" (UID: \"e85be7c5-53ae-4992-a98b-b1842269bb42\") "
Oct 30 21:24:44.923363 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:24:44.923307    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e85be7c5-53ae-4992-a98b-b1842269bb42-maas-api-openshift-ca" pod "e85be7c5-53ae-4992-a98b-b1842269bb42" (UID: "e85be7c5-53ae-4992-a98b-b1842269bb42"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:24:44.924909 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:24:44.924887    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e85be7c5-53ae-4992-a98b-b1842269bb42-maas-api-xks-ca" pod "e85be7c5-53ae-4992-a98b-b1842269bb42" (UID: "e85be7c5-53ae-4992-a98b-b1842269bb42"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:24:44.925015 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:24:44.924999    4521 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e85be7c5-53ae-4992-a98b-b1842269bb42-kube-api-access-ggsgc" pod "e85be7c5-53ae-4992-a98b-b1842269bb42" (UID: "e85be7c5-53ae-4992-a98b-b1842269bb42"). InnerVolumeSpecName "kube-api-access-ggsgc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:24:45.024070 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:24:45.024033    4521 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e85be7c5-53ae-4992-a98b-b1842269bb42-maas-api-openshift-ca\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:24:45.024070 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:24:45.024066    4521 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggsgc\" (UniqueName: \"kubernetes.io/projected/e85be7c5-53ae-4992-a98b-b1842269bb42-kube-api-access-ggsgc\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:24:45.024070 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:24:45.024078    4521 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e85be7c5-53ae-4992-a98b-b1842269bb42-maas-api-xks-ca\") on node \"ip-10-0-131-125.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:24:45.682935 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:24:45.682904    4521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-lm4gg"
Oct 30 21:24:45.683103 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:24:45.682908    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-lm4gg" event={"ID":"e85be7c5-53ae-4992-a98b-b1842269bb42","Type":"ContainerDied","Data":"cd8b0059bc241ec63f827f4819b2d74ebc4e5a1385d1c41fa6a9b29367c2361c"}
Oct 30 21:24:45.683103 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:24:45.683035    4521 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cd8b0059bc241ec63f827f4819b2d74ebc4e5a1385d1c41fa6a9b29367c2361c"
Oct 30 21:24:46.428914 ip-10-0-131-125 kubenswrapper[4521]: E1030 21:24:46.428879    4521 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lkn2k" containerName="extractor"
Oct 30 21:24:52.376240 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:24:52.376199    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f4kxb"]
Oct 30 21:25:04.030474 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:25:04.030439    4521 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-vgh86_f451091e-4af7-412a-8e0b-8cc42c1726ec/manager/0.log"
Oct 30 21:25:04.247867 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:25:04.247828    4521 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-vtzgz_d306b705-56e3-4de0-a4cb-fe23b9621cf0/manager/0.log"
Oct 30 21:25:06.238874 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:25:06.238838    4521 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-tvkpg_0549520a-091d-4e1c-bc20-ea430909f25e/revoke-keys/0.log"
Oct 30 21:25:06.789483 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:25:06.789452    4521 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-lm4gg_e85be7c5-53ae-4992-a98b-b1842269bb42/revoke-keys/0.log"
Oct 30 21:25:07.163827 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:25:07.163744    4521 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-nsg9r_0a95d0b7-c673-45ee-b23d-e958e79a80a5/revoke-keys/0.log"
Oct 30 21:25:09.133414 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:25:09.133381    4521 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-x9bck_8235f71b-607f-4cd7-a69a-de5b52fa90cd/manager/0.log"
Oct 30 21:25:09.320705 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:25:09.320675    4521 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-f97mf_a78e2ba4-f4b7-4df7-9e8e-e7cf0e717c42/manager/0.log"
Oct 30 21:25:09.715750 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:25:09.715722    4521 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-2zlw7_0c31240f-bde3-4853-a2c3-6f83ab926529/manager/0.log"
Oct 30 21:25:09.934665 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:25:09.934630    4521 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-f4kxb_3d5dbf9c-8743-4193-8c60-3f49a346776d/limitador/0.log"
Oct 30 21:25:10.146199 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:25:10.146113    4521 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-fz9qs_91b6f058-7deb-4263-baae-9ed0ad3e7380/manager/0.log"
Oct 30 21:25:10.943065 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:25:10.943029    4521 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-c574b7b6f-7jhkd_8eba7406-6c06-4fa4-8022-0de75cfdbeb6/istio-proxy/0.log"
Oct 30 21:25:12.269652 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:25:12.269619    4521 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58c6585447-bckjb_62149acf-8605-4f8d-b1ac-6e8d6b4865fb/istio-proxy/1.log"
Oct 30 21:25:12.455501 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:25:12.455474    4521 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58c6585447-fw7lj_5311e7d9-ac35-419c-a7d8-2bceba0f4f44/istio-proxy/1.log"
Oct 30 21:25:13.028031 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:25:13.028000    4521 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58c6585447-mgx2j_3f51564c-e79a-437a-9e62-3b2750e0916a/istio-proxy/1.log"
Oct 30 21:25:13.584238 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:25:13.584206    4521 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-bc8769f9-rxf86_f421589f-1efe-4427-a723-435502b4a14b/istio-proxy/0.log"
Oct 30 21:25:14.732423 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:25:14.732393    4521 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58c6585447-bckjb_62149acf-8605-4f8d-b1ac-6e8d6b4865fb/istio-proxy/0.log"
Oct 30 21:25:14.916338 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:25:14.916299    4521 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58c6585447-fw7lj_5311e7d9-ac35-419c-a7d8-2bceba0f4f44/istio-proxy/0.log"
Oct 30 21:25:15.286778 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:25:15.286745    4521 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58c6585447-mgx2j_3f51564c-e79a-437a-9e62-3b2750e0916a/istio-proxy/0.log"
Oct 30 21:25:16.428946 ip-10-0-131-125 kubenswrapper[4521]: E1030 21:25:16.428912    4521 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lkn2k" containerName="extractor"
Oct 30 21:25:28.206194 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:25:28.206162    4521 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-vgh86_f451091e-4af7-412a-8e0b-8cc42c1726ec/manager/0.log"
Oct 30 21:25:28.410030 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:25:28.409996    4521 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-vtzgz_d306b705-56e3-4de0-a4cb-fe23b9621cf0/manager/0.log"
Oct 30 21:25:30.161961 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:25:30.161933    4521 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-tvkpg_0549520a-091d-4e1c-bc20-ea430909f25e/revoke-keys/0.log"
Oct 30 21:25:30.518977 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:25:30.518948    4521 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-lm4gg_e85be7c5-53ae-4992-a98b-b1842269bb42/revoke-keys/0.log"
Oct 30 21:25:30.869087 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:25:30.868998    4521 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-nsg9r_0a95d0b7-c673-45ee-b23d-e958e79a80a5/revoke-keys/0.log"
Oct 30 21:25:33.455955 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:25:33.455920    4521 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-x9bck_8235f71b-607f-4cd7-a69a-de5b52fa90cd/manager/0.log"
Oct 30 21:25:33.633430 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:25:33.633394    4521 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-f97mf_a78e2ba4-f4b7-4df7-9e8e-e7cf0e717c42/manager/0.log"
Oct 30 21:25:34.031109 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:25:34.031079    4521 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-2zlw7_0c31240f-bde3-4853-a2c3-6f83ab926529/manager/0.log"
Oct 30 21:25:34.223549 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:25:34.223518    4521 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-f4kxb_3d5dbf9c-8743-4193-8c60-3f49a346776d/limitador/0.log"
Oct 30 21:25:34.419802 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:25:34.419715    4521 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-fz9qs_91b6f058-7deb-4263-baae-9ed0ad3e7380/manager/0.log"
Oct 30 21:25:35.007197 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:25:35.007168    4521 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-c574b7b6f-7jhkd_8eba7406-6c06-4fa4-8022-0de75cfdbeb6/istio-proxy/0.log"
Oct 30 21:25:36.361175 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:25:36.361149    4521 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58c6585447-bckjb_62149acf-8605-4f8d-b1ac-6e8d6b4865fb/istio-proxy/1.log"
Oct 30 21:25:36.578825 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:25:36.578795    4521 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58c6585447-fw7lj_5311e7d9-ac35-419c-a7d8-2bceba0f4f44/istio-proxy/1.log"
Oct 30 21:25:37.214997 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:25:37.214947    4521 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58c6585447-mgx2j_3f51564c-e79a-437a-9e62-3b2750e0916a/istio-proxy/1.log"
Oct 30 21:25:37.774807 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:25:37.774774    4521 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-bc8769f9-rxf86_f421589f-1efe-4427-a723-435502b4a14b/istio-proxy/0.log"
Oct 30 21:25:46.428654 ip-10-0-131-125 kubenswrapper[4521]: E1030 21:25:46.428610    4521 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lkn2k" containerName="extractor"
Oct 30 21:25:54.879224 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:25:54.879191    4521 scope.go:122] "RemoveContainer" containerID="c8e831d9fa0b8730ff0fbdfde3da0920a95db2c629cb231e481ec2fbfe25679d"
Oct 30 21:25:54.888091 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:25:54.888071    4521 scope.go:122] "RemoveContainer" containerID="e19f9a9def550c48fdcf7accf8fda4d13b6ef8ef72945e796c212fe0f6366264"
Oct 30 21:26:16.428786 ip-10-0-131-125 kubenswrapper[4521]: E1030 21:26:16.428701    4521 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lkn2k" containerName="extractor"
Oct 30 21:26:46.429113 ip-10-0-131-125 kubenswrapper[4521]: E1030 21:26:46.429072    4521 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lkn2k" containerName="extractor"
Oct 30 21:27:07.708883 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:27:07.708845    4521 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-dwnkd/must-gather-5v2fb"]
Oct 30 21:27:07.709308 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:27:07.709214    4521 cpu_manager.go:414] "containerMap: removing container" podUID="e85be7c5-53ae-4992-a98b-b1842269bb42" containerName="revoke-keys"
Oct 30 21:27:07.709308 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:27:07.709225    4521 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e85be7c5-53ae-4992-a98b-b1842269bb42" containerName="revoke-keys"
Oct 30 21:27:07.709308 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:27:07.709293    4521 memory_manager.go:365] "RemoveStaleState removing state" podUID="e85be7c5-53ae-4992-a98b-b1842269bb42" containerName="revoke-keys"
Oct 30 21:27:07.712732 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:27:07.712715    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dwnkd/must-gather-5v2fb"
Oct 30 21:27:07.721392 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:27:07.721372    4521 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-dwnkd\"/\"kube-root-ca.crt\""
Oct 30 21:27:07.722434 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:27:07.722413    4521 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-dwnkd\"/\"openshift-service-ca.crt\""
Oct 30 21:27:07.722533 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:27:07.722431    4521 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-dwnkd\"/\"default-dockercfg-h2dpm\""
Oct 30 21:27:07.756938 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:27:07.756896    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dwnkd/must-gather-5v2fb"]
Oct 30 21:27:07.797591 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:27:07.797559    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b7c29b37-f91c-4097-b1c2-536d915a8750-must-gather-output\") pod \"must-gather-5v2fb\" (UID: \"b7c29b37-f91c-4097-b1c2-536d915a8750\") " pod="openshift-must-gather-dwnkd/must-gather-5v2fb"
Oct 30 21:27:07.797766 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:27:07.797619    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgqxc\" (UniqueName: \"kubernetes.io/projected/b7c29b37-f91c-4097-b1c2-536d915a8750-kube-api-access-pgqxc\") pod \"must-gather-5v2fb\" (UID: \"b7c29b37-f91c-4097-b1c2-536d915a8750\") " pod="openshift-must-gather-dwnkd/must-gather-5v2fb"
Oct 30 21:27:07.898538 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:27:07.898496    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgqxc\" (UniqueName: \"kubernetes.io/projected/b7c29b37-f91c-4097-b1c2-536d915a8750-kube-api-access-pgqxc\") pod \"must-gather-5v2fb\" (UID: \"b7c29b37-f91c-4097-b1c2-536d915a8750\") " pod="openshift-must-gather-dwnkd/must-gather-5v2fb"
Oct 30 21:27:07.898729 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:27:07.898567    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b7c29b37-f91c-4097-b1c2-536d915a8750-must-gather-output\") pod \"must-gather-5v2fb\" (UID: \"b7c29b37-f91c-4097-b1c2-536d915a8750\") " pod="openshift-must-gather-dwnkd/must-gather-5v2fb"
Oct 30 21:27:07.898889 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:27:07.898874    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b7c29b37-f91c-4097-b1c2-536d915a8750-must-gather-output\") pod \"must-gather-5v2fb\" (UID: \"b7c29b37-f91c-4097-b1c2-536d915a8750\") " pod="openshift-must-gather-dwnkd/must-gather-5v2fb"
Oct 30 21:27:07.913117 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:27:07.913084    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgqxc\" (UniqueName: \"kubernetes.io/projected/b7c29b37-f91c-4097-b1c2-536d915a8750-kube-api-access-pgqxc\") pod \"must-gather-5v2fb\" (UID: \"b7c29b37-f91c-4097-b1c2-536d915a8750\") " pod="openshift-must-gather-dwnkd/must-gather-5v2fb"
Oct 30 21:27:08.027124 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:27:08.027040    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dwnkd/must-gather-5v2fb"
Oct 30 21:27:08.306039 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:27:08.306015    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dwnkd/must-gather-5v2fb"]
Oct 30 21:27:08.308079 ip-10-0-131-125 kubenswrapper[4521]: W1030 21:27:08.308049    4521 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb7c29b37_f91c_4097_b1c2_536d915a8750.slice/crio-3bbdd622d49d52d9dd0735a927f44cd533fad331f1dbc7876cfce6d413d7de70 WatchSource:0}: Error finding container 3bbdd622d49d52d9dd0735a927f44cd533fad331f1dbc7876cfce6d413d7de70: Status 404 returned error can't find the container with id 3bbdd622d49d52d9dd0735a927f44cd533fad331f1dbc7876cfce6d413d7de70
Oct 30 21:27:08.309962 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:27:08.309947    4521 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 21:27:08.434107 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:27:08.434067    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dwnkd/must-gather-5v2fb" event={"ID":"b7c29b37-f91c-4097-b1c2-536d915a8750","Type":"ContainerStarted","Data":"3bbdd622d49d52d9dd0735a927f44cd533fad331f1dbc7876cfce6d413d7de70"}
Oct 30 21:27:09.449133 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:27:09.449091    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dwnkd/must-gather-5v2fb" event={"ID":"b7c29b37-f91c-4097-b1c2-536d915a8750","Type":"ContainerStarted","Data":"8241b00bbc0a9512d17435424e95d2da6d4f83ef623cea296baff538c83eb23c"}
Oct 30 21:27:10.456745 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:27:10.456710    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dwnkd/must-gather-5v2fb" event={"ID":"b7c29b37-f91c-4097-b1c2-536d915a8750","Type":"ContainerStarted","Data":"bf4e796ac96174c11201ae7e9472a8bb10380b413a82bb839ad3f7cd4c372525"}
Oct 30 21:27:10.478170 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:27:10.477053    4521 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-dwnkd/must-gather-5v2fb" podStartSLOduration=2.510042781 podStartE2EDuration="3.477034179s" totalImagesPullingTime="966.991398ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:27:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:27:10.475119469 +0000 UTC m=+2897.820740707" watchObservedRunningTime="2026-10-30 21:27:10.477034179 +0000 UTC m=+2897.822655416"
Oct 30 21:27:11.343411 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:27:11.343317    4521 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-xkfsj_a03cdbf5-91a2-41eb-8c1e-3c0f40da2632/global-pull-secret-syncer/0.log"
Oct 30 21:27:11.623192 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:27:11.623110    4521 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-qcrpl_d585cbd8-cb89-4718-9af4-716f64cc2598/konnectivity-agent/0.log"
Oct 30 21:27:11.647524 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:27:11.647496    4521 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-131-125.us-east-2.compute.internal_cbbd659d200641ef129d88b1e4575eca/haproxy/0.log"
Oct 30 21:27:16.450233 ip-10-0-131-125 kubenswrapper[4521]: E1030 21:27:16.450198    4521 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lkn2k" containerName="extractor"
Oct 30 21:27:18.557474 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:27:18.557435    4521 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-x9bck_8235f71b-607f-4cd7-a69a-de5b52fa90cd/manager/0.log"
Oct 30 21:27:18.591764 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:27:18.591725    4521 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-f97mf_a78e2ba4-f4b7-4df7-9e8e-e7cf0e717c42/manager/0.log"
Oct 30 21:27:18.695389 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:27:18.695288    4521 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-2zlw7_0c31240f-bde3-4853-a2c3-6f83ab926529/manager/0.log"
Oct 30 21:27:18.725989 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:27:18.725952    4521 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-f4kxb_3d5dbf9c-8743-4193-8c60-3f49a346776d/limitador/0.log"
Oct 30 21:27:18.808215 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:27:18.808111    4521 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-fz9qs_91b6f058-7deb-4263-baae-9ed0ad3e7380/manager/0.log"
Oct 30 21:27:20.494080 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:27:20.494047    4521 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5s5hx_df915ea2-605d-413e-8942-083da7b4ba8f/node-exporter/0.log"
Oct 30 21:27:20.525576 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:27:20.525543    4521 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5s5hx_df915ea2-605d-413e-8942-083da7b4ba8f/kube-rbac-proxy/0.log"
Oct 30 21:27:20.555355 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:27:20.552374    4521 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5s5hx_df915ea2-605d-413e-8942-083da7b4ba8f/init-textfile/0.log"
Oct 30 21:27:20.712228 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:27:20.712186    4521 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-dwnkd/perf-node-gather-daemonset-9hc9h"]
Oct 30 21:27:20.719673 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:27:20.719644    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dwnkd/perf-node-gather-daemonset-9hc9h"
Oct 30 21:27:20.727091 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:27:20.727057    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dwnkd/perf-node-gather-daemonset-9hc9h"]
Oct 30 21:27:20.729982 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:27:20.729951    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rb79\" (UniqueName: \"kubernetes.io/projected/6fc51681-d25f-4445-aaff-785fe359eca1-kube-api-access-5rb79\") pod \"perf-node-gather-daemonset-9hc9h\" (UID: \"6fc51681-d25f-4445-aaff-785fe359eca1\") " pod="openshift-must-gather-dwnkd/perf-node-gather-daemonset-9hc9h"
Oct 30 21:27:20.730617 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:27:20.730592    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6fc51681-d25f-4445-aaff-785fe359eca1-proc\") pod \"perf-node-gather-daemonset-9hc9h\" (UID: \"6fc51681-d25f-4445-aaff-785fe359eca1\") " pod="openshift-must-gather-dwnkd/perf-node-gather-daemonset-9hc9h"
Oct 30 21:27:20.730785 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:27:20.730769    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6fc51681-d25f-4445-aaff-785fe359eca1-sys\") pod \"perf-node-gather-daemonset-9hc9h\" (UID: \"6fc51681-d25f-4445-aaff-785fe359eca1\") " pod="openshift-must-gather-dwnkd/perf-node-gather-daemonset-9hc9h"
Oct 30 21:27:20.730932 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:27:20.730906    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6fc51681-d25f-4445-aaff-785fe359eca1-podres\") pod \"perf-node-gather-daemonset-9hc9h\" (UID: \"6fc51681-d25f-4445-aaff-785fe359eca1\") " pod="openshift-must-gather-dwnkd/perf-node-gather-daemonset-9hc9h"
Oct 30 21:27:20.731013 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:27:20.730957    4521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6fc51681-d25f-4445-aaff-785fe359eca1-lib-modules\") pod \"perf-node-gather-daemonset-9hc9h\" (UID: \"6fc51681-d25f-4445-aaff-785fe359eca1\") " pod="openshift-must-gather-dwnkd/perf-node-gather-daemonset-9hc9h"
Oct 30 21:27:20.831946 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:27:20.831852    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rb79\" (UniqueName: \"kubernetes.io/projected/6fc51681-d25f-4445-aaff-785fe359eca1-kube-api-access-5rb79\") pod \"perf-node-gather-daemonset-9hc9h\" (UID: \"6fc51681-d25f-4445-aaff-785fe359eca1\") " pod="openshift-must-gather-dwnkd/perf-node-gather-daemonset-9hc9h"
Oct 30 21:27:20.831946 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:27:20.831918    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6fc51681-d25f-4445-aaff-785fe359eca1-proc\") pod \"perf-node-gather-daemonset-9hc9h\" (UID: \"6fc51681-d25f-4445-aaff-785fe359eca1\") " pod="openshift-must-gather-dwnkd/perf-node-gather-daemonset-9hc9h"
Oct 30 21:27:20.832186 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:27:20.832061    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6fc51681-d25f-4445-aaff-785fe359eca1-proc\") pod \"perf-node-gather-daemonset-9hc9h\" (UID: \"6fc51681-d25f-4445-aaff-785fe359eca1\") " pod="openshift-must-gather-dwnkd/perf-node-gather-daemonset-9hc9h"
Oct 30 21:27:20.832186 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:27:20.832120    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6fc51681-d25f-4445-aaff-785fe359eca1-sys\") pod \"perf-node-gather-daemonset-9hc9h\" (UID: \"6fc51681-d25f-4445-aaff-785fe359eca1\") " pod="openshift-must-gather-dwnkd/perf-node-gather-daemonset-9hc9h"
Oct 30 21:27:20.832300 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:27:20.832207    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6fc51681-d25f-4445-aaff-785fe359eca1-podres\") pod \"perf-node-gather-daemonset-9hc9h\" (UID: \"6fc51681-d25f-4445-aaff-785fe359eca1\") " pod="openshift-must-gather-dwnkd/perf-node-gather-daemonset-9hc9h"
Oct 30 21:27:20.832300 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:27:20.832249    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6fc51681-d25f-4445-aaff-785fe359eca1-sys\") pod \"perf-node-gather-daemonset-9hc9h\" (UID: \"6fc51681-d25f-4445-aaff-785fe359eca1\") " pod="openshift-must-gather-dwnkd/perf-node-gather-daemonset-9hc9h"
Oct 30 21:27:20.832300 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:27:20.832251    4521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6fc51681-d25f-4445-aaff-785fe359eca1-lib-modules\") pod \"perf-node-gather-daemonset-9hc9h\" (UID: \"6fc51681-d25f-4445-aaff-785fe359eca1\") " pod="openshift-must-gather-dwnkd/perf-node-gather-daemonset-9hc9h"
Oct 30 21:27:20.832496 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:27:20.832337    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6fc51681-d25f-4445-aaff-785fe359eca1-lib-modules\") pod \"perf-node-gather-daemonset-9hc9h\" (UID: \"6fc51681-d25f-4445-aaff-785fe359eca1\") " pod="openshift-must-gather-dwnkd/perf-node-gather-daemonset-9hc9h"
Oct 30 21:27:20.832496 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:27:20.832378    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6fc51681-d25f-4445-aaff-785fe359eca1-podres\") pod \"perf-node-gather-daemonset-9hc9h\" (UID: \"6fc51681-d25f-4445-aaff-785fe359eca1\") " pod="openshift-must-gather-dwnkd/perf-node-gather-daemonset-9hc9h"
Oct 30 21:27:20.844420 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:27:20.844388    4521 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rb79\" (UniqueName: \"kubernetes.io/projected/6fc51681-d25f-4445-aaff-785fe359eca1-kube-api-access-5rb79\") pod \"perf-node-gather-daemonset-9hc9h\" (UID: \"6fc51681-d25f-4445-aaff-785fe359eca1\") " pod="openshift-must-gather-dwnkd/perf-node-gather-daemonset-9hc9h"
Oct 30 21:27:21.039057 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:27:21.039011    4521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dwnkd/perf-node-gather-daemonset-9hc9h"
Oct 30 21:27:21.223723 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:27:21.223693    4521 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-9tfw6_f7db4cd9-6dc6-4201-9c54-3c52be1c889e/prometheus-operator-admission-webhook/0.log"
Oct 30 21:27:21.352854 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:27:21.343730    4521 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dwnkd/perf-node-gather-daemonset-9hc9h"]
Oct 30 21:27:21.567936 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:27:21.567896    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dwnkd/perf-node-gather-daemonset-9hc9h" event={"ID":"6fc51681-d25f-4445-aaff-785fe359eca1","Type":"ContainerStarted","Data":"140786490ed4580b620e8c8872810ac4f0ce32b16ff92878483c2b9b6e8c0c26"}
Oct 30 21:27:21.568417 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:27:21.567961    4521 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dwnkd/perf-node-gather-daemonset-9hc9h" event={"ID":"6fc51681-d25f-4445-aaff-785fe359eca1","Type":"ContainerStarted","Data":"8894e6cdae3583b0af2a996cf3eb25c6c1d60318fb55ba0a1a5a89d87131bb30"}
Oct 30 21:27:21.568417 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:27:21.567993    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-dwnkd/perf-node-gather-daemonset-9hc9h"
Oct 30 21:27:21.589675 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:27:21.589611    4521 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-dwnkd/perf-node-gather-daemonset-9hc9h" podStartSLOduration=1.589589811 podStartE2EDuration="1.589589811s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:27:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:27:21.589500295 +0000 UTC m=+2908.935121533" watchObservedRunningTime="2026-10-30 21:27:21.589589811 +0000 UTC m=+2908.935211067"
Oct 30 21:27:23.869010 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:27:23.868981    4521 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-6cf8db96c7-kw294_9241b135-3961-4ef5-9fcd-96a078e2475f/console/0.log"
Oct 30 21:27:23.908599 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:27:23.908552    4521 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6dd789c8b7-ms68j_426b83f8-572e-4178-adb4-bf1c2a6bee2b/download-server/0.log"
Oct 30 21:27:25.770425 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:27:25.770397    4521 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-fblmm_30561fcc-3184-4116-b179-18941ff26a20/dns/0.log"
Oct 30 21:27:25.794992 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:27:25.794965    4521 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-fblmm_30561fcc-3184-4116-b179-18941ff26a20/kube-rbac-proxy/0.log"
Oct 30 21:27:25.881915 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:27:25.881889    4521 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-4kpcm_c6467dfc-7d9c-4ba8-ac2e-2ce19a74a0d4/dns-node-resolver/0.log"
Oct 30 21:27:26.580129 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:27:26.580097    4521 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-7579767596-gmrh5_ec3ad2a6-6025-4187-8bac-1ab4d7ea5de0/registry/0.log"
Oct 30 21:27:26.608030 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:27:26.608001    4521 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-2kdx9_880d0493-e39d-4d9c-b58b-aa48c33aeaef/node-ca/0.log"
Oct 30 21:27:27.596053 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:27:27.596023    4521 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-dwnkd/perf-node-gather-daemonset-9hc9h"
Oct 30 21:27:27.817428 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:27:27.817390    4521 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-c574b7b6f-7jhkd_8eba7406-6c06-4fa4-8022-0de75cfdbeb6/istio-proxy/0.log"
Oct 30 21:27:30.433594 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:27:30.433556    4521 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58c6585447-bckjb_62149acf-8605-4f8d-b1ac-6e8d6b4865fb/istio-proxy/0.log"
Oct 30 21:27:30.434112 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:27:30.433694    4521 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58c6585447-bckjb_62149acf-8605-4f8d-b1ac-6e8d6b4865fb/istio-proxy/1.log"
Oct 30 21:27:30.472445 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:27:30.472412    4521 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58c6585447-fw7lj_5311e7d9-ac35-419c-a7d8-2bceba0f4f44/istio-proxy/1.log"
Oct 30 21:27:30.472676 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:27:30.472506    4521 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58c6585447-fw7lj_5311e7d9-ac35-419c-a7d8-2bceba0f4f44/istio-proxy/0.log"
Oct 30 21:27:30.598537 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:27:30.598496    4521 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58c6585447-mgx2j_3f51564c-e79a-437a-9e62-3b2750e0916a/istio-proxy/1.log"
Oct 30 21:27:30.598869 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:27:30.598852    4521 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58c6585447-mgx2j_3f51564c-e79a-437a-9e62-3b2750e0916a/istio-proxy/0.log"
Oct 30 21:27:30.701151 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:27:30.701102    4521 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-bc8769f9-rxf86_f421589f-1efe-4427-a723-435502b4a14b/istio-proxy/0.log"
Oct 30 21:27:31.242156 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:27:31.242121    4521 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-c574b7b6f-7jhkd_8eba7406-6c06-4fa4-8022-0de75cfdbeb6/istio-proxy/0.log"
Oct 30 21:27:32.236596 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:27:32.236570    4521 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-q2wtm_747c15be-3782-4065-a6dc-73d9fe7a4b4f/serve-healthcheck-canary/0.log"
Oct 30 21:27:33.102660 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:27:33.102620    4521 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-lkn2k_451ecbeb-c188-4e49-8e4e-08af1051ae0f/kube-rbac-proxy/0.log"
Oct 30 21:27:33.130244 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:27:33.130210    4521 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-lkn2k_451ecbeb-c188-4e49-8e4e-08af1051ae0f/exporter/0.log"
Oct 30 21:27:33.161930 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:27:33.161899    4521 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-lkn2k_451ecbeb-c188-4e49-8e4e-08af1051ae0f/extractor/0.log"
Oct 30 21:27:33.202576 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:27:33.202513    4521 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_periodic-gathering-stq6m-cr8z8_4bb3e3ca-18ce-4d77-9150-30ca3adfc11f/insights-gathering/0.log"
Oct 30 21:27:33.648347 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:27:33.648257    4521 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58c6585447-bckjb_62149acf-8605-4f8d-b1ac-6e8d6b4865fb/istio-proxy/0.log"
Oct 30 21:27:33.649569 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:27:33.649548    4521 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58c6585447-bckjb_62149acf-8605-4f8d-b1ac-6e8d6b4865fb/istio-proxy/1.log"
Oct 30 21:27:33.667635 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:27:33.667606    4521 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58c6585447-fw7lj_5311e7d9-ac35-419c-a7d8-2bceba0f4f44/istio-proxy/0.log"
Oct 30 21:27:33.667781 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:27:33.667705    4521 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58c6585447-fw7lj_5311e7d9-ac35-419c-a7d8-2bceba0f4f44/istio-proxy/1.log"
Oct 30 21:27:33.721570 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:27:33.721536    4521 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58c6585447-mgx2j_3f51564c-e79a-437a-9e62-3b2750e0916a/istio-proxy/1.log"
Oct 30 21:27:33.721953 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:27:33.721935    4521 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58c6585447-mgx2j_3f51564c-e79a-437a-9e62-3b2750e0916a/istio-proxy/0.log"
Oct 30 21:27:33.767673 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:27:33.767647    4521 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-bc8769f9-rxf86_f421589f-1efe-4427-a723-435502b4a14b/istio-proxy/0.log"
Oct 30 21:27:35.741936 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:27:35.741854    4521 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-c574b7b6f-7jhkd_8eba7406-6c06-4fa4-8022-0de75cfdbeb6/istio-proxy/0.log"
Oct 30 21:27:35.877173 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:27:35.877142    4521 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-5d46577d6f-rvfc7_f09f227c-2059-4bb6-af30-f18c4c261487/manager/0.log"
Oct 30 21:27:35.897948 ip-10-0-131-125 kubenswrapper[4521]: I1030 21:27:35.897919    4521 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-5d46577d6f-rvfc7_f09f227c-2059-4bb6-af30-f18c4c261487/copy-manifests/0.log"